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SUMMARY 


This  report  documents  1)  an  analysis  of  feasibility  and  2) 
the  development  of  a  preliminary  structure  for  a  computerized 
data  base  on  capability  and  availability  of  computer-based 
instruction  (CBI)  technologies  within  the  Department  of  Defense 
(DOD).  it  concludes  that  the  proposed  data  base,  although 
logistically  complex,  can  be  developed  and  would  be  of 
considerable  value  in  providing  ready  access  to  technology 
information  for  a  variety  of  DOD  users.  Rationale  for 
recommendations  is  based  on  review  and  synthesis  of  the  CBI 
research  and  development  (R&D)  literature,  structured  interviews 
with  a  cross-section  of  potential  data  base  users,  and  an 
analysis  of  existing  R&D  data  bases.  This  report  provides  a 
first  step  in  the  data  base  development  process,  that  of 
examining  the  user  requirements,  issues  of  importance,  and 
potential  development  considerations.  It  also  describes  the 
subsequent  requirements  leading  to  creation  of  a  user-oriented 
R&D  data  base  for  CBI  technology. 

Discussions  of  the  problem  introduce  the  concept  of  a  "CBI 
applications  gap"  due  to  rapid  but  uneven  growth  of  the 
technology,  and  a  resultant  confusion  by  potential  users 
concerning  availability  and  characteristics  of  new  CBI  methods. 
A  brief  review  of  CBI  literature  is  presented,  along  with 
indications  of  general  trends  in  CBI  research  and  development 
and  some  assessments  of  high-payoff  directions  for  the 
technology.  Reviews,  trends  and  assessments  are  heavily 
user-oriented,  with  emphasis  on  assessing  the  impact  of 
technology  innovation  on  users  and  defining  user  requirements 
for  technology  information.  Interviews  with  CBI  developers  and 
other  members  of  the  user  community  were  conducted.  Inputs  were 
solicited  on  both  practicality  and  value  of  the  proposed  data 
base  and  on  important  ongoing  technology  efforts  expected  to  be 
complete  in  the  near  future.  Results  of  the  survey  are 
generally  consistent  with  findings  from  the  literature  analysis 
and  support  the  need  for  a  technology  data  base. 
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Characteristics  of  alternative  data  bases  are  presented  and 
related  to  user  requirements.  It  is  concluded  that  a  data  base 
structured  along  the  lines  of  a  "consumers  guide"  to  available 
technology  is  of  greatest  potential  value.  The  process  of 
providing  such  a  data  base  goes  beyond  the  simple  compilation  of 
available  technology  information.  From  the  analysis,  two 
important  issues  were  identified.  The  first  addresses  the  need 
for  specific  attention  to  the  characteristics  and  data  needs  of 
potential  users  of  both  CBI  technology  and  the  proposed  data 
base.  The  second  is  concerned  with  the  requirement  for 
taxonomically  sound  classification  schema  to  aid  in  collection 
of  information  for  the  data  base.  The  schema  should  describe 
features  of  CBI  systems  in  a  form  that  generates  data  storage 
and  retrieval  cues.  A  "strawman"  version  of  the  schema  is 
presented,  which  categorizes  technology  information  in  terms  of 
Functions  Performed,  System  Characteristics,  Training  Outcomes 
and  Area  of  Application.  The  impact  of  these  two  issues  on  the 
design  and  functioning  of  the  data  base  are  examined  and 
described. 
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SECTION  I 

INTRODUCTION  AND  BACKGROUND 


There  is  a  nearly  universal  belief  that  military  training 
needs  to  be  improved.  This  belief  is  shared  throughout  the 
diversity  of  the  training  community,  by  individuals  at  every 
echelon  of  involvement.  The  problem  has  been  articulated  by 
instructors,  instructional  planners,  managers,  researchers  and 
even  trainees.  It  is  recognized  by  groups  who  may  agree  on 
little  else  (almost  certainly  not  the  solution),  but  who  share  a 
common  understanding  of  the  importance  of  operational  readiness 
and  its  linkages  to  the  adequacy  and  effectiveness  of  training. 
The  demands  placed  on  the  training  process  have  increased 
greatly  with  the  advances  in  operational  equipment 
sophistication.  New  weapon  systems  require  operators  and 
maintainers  to  possess  a  steadily  larger  body  of  technical 
information  and  specific  technical  skills.  Providing  training 
for  these  requirements  via  traditional  instructional  methods  has 
become  increasingly  difficult.  Conveying  the  information  and 
skills  required  to  operate  technologically  advanced  equipment 
will  ultimately  require  an  equivalent  sophistication  in  the 
technology  of  training. 

Among  the  training  technologies  responding  to  these 
requirements  is  the  class  of  instructional  development,  delivery 
and  management  techniques  collectively  referred  to  as 
computer-based  instruction  (CBI).  Although  CBI  has  yet  to 
realize  the  full  measure  of  contributions  expected  from  the 
technology  for  the  last  two  decades  (Kearsley,  Hunter  &  Seidel, 
1983;  Suppes,  1979),  it  continues  to  offer  a  powerful 
opportunity  for  improvements  in  the  effectiveness  and  cost  of 
training.  There  has  been  concern,  however,  that  the  numbers, 
complexity  and  diversity  of  available  CBI  methods  have  become  a 
source  of  considerable  uncertainty  in  selecting  the  most 
appropriate  method  of  CBI  technology  for  a  specific 


i 


application.  This  is  to  a  considerable  extent  attributable  to 
the  lack  of  attention  paid  by  technology  developers  and 
docuraenters  to  the  issue  of  the  intended  end-user  of  a  product, 
the  "consumer"  of  CBI.  Although  there  is  considerable 
discussion  of  the  need  to  serve  the  "user,”  it  is  not  always 
clear  who  the  intended  user  is  or  where  he  resides  in  the 
training  process.  Users  can  become  involved  in  the  training 
process  during  any  of  several  entry  points,  including 
preliminary  decisions  (programming  and  budgeting),  technology 
development,  instructional  development,  and  instructional 
delivery  and  revision.  Each  of  these  end-users  is  legitimate 
and  each  can  require  different  types  of  data  about  the 
technology,  at  varying  levels  of  specificity  and  detail. 

Just  as  a  military  system  designed  without  a  clear  picture 
of  the  skills  and  experience  expected  from  the  intended  operator 
is  likely  to  exceed  the  capabilities  of  available  personnel,  so 
an  instructional  support  methodology  without  a  clear  definition 
of  the  training  and  background  of  the  "user"  may  require  an 
extended  team  of  superbeings  to  realize  its  full  potential.  As 
Montgomery  and  Judd  (1979)  pointed  out,  instructional 
development  with  such  systems  requires  "...a  versatile 
professor,  an  expert  in  the  subject  matter,  an  experienced 
teacher  with  sound  but  innovative  ideas  about  instructional 
presentation,  and  a  capable  programmer"  (p.  20),  preferably 
combined  into  a  single  individual.  Amidst  this  uncertainty 
about  end-users,  the  potential  value  of  CBI  is  reduced  by  a  lack 
of  understandable  information  on  the  availability  and 
characteristics  of  specific  CBI  technologies  and  by  consequent 
tendencies  toward  inappropriate  technology  use. 

Confusion  about  intended  user  characteristics  has  been 
aggravated  by  dramatic  changes  in  affordability  of  small 
computers.  While  availability  of  powerful,  inexpensive 
processing  has  broadened  the  user  base  for  CBI,  it  has  at  the 
same  time  brought  about  a  rapid  expansion  of  languages, 
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architectures  and  authoring  approaches  which  can  seriously 
confuse  potential  entrants  into  the  user  community  who  lack 
formal  training  in  these  new  disciplines. 

These  "growing  pains"  and  their  impact  on  the  growth 
potential  of  CBI  in  military  training  are  discussed  in  detail  by 
Funaro  and  Lane  (1985).  They  address  the  problem  of  the  CBI 
applications  lag  in  the  more  general  context  of  introducing 
technology  into  systems,  and  suggest  that  periods  of  rapid 
technology  growth  are  likely  to  cause  a  slowdown  in  applications 
while  users  wait  for  the  technology  to  stabilize.  This  process 
is  avoidable  only  by  some  deliberate  intervention  to  counteract 
user  uncertainty.  Among  the  intervention  actions  recommended 
are  an  explicit  focus  on  the  user  or  "consumer"  of  CBI  in  the 
form  of  increased  availability  of  technology  data  and  the 
systematic  use  of  technology  demonstrations  to  reduce  user 
concerns  about  technical  risks.  They  conclude  that: 

•  Assistance  must  be  provided  to  the  CBI  user  in  matching 
his  particular  needs  to  the  available  technology  (a  "consumers 
guide  to  CBI" ) . 

•  To  the  fullest  extent  possible,  technology  should  be 
standardized  across  the  services  and  common  definitions 
developed. 

•  Realistic  demonstrations  of  CBI  as  a  viable  method  of 
solving  practical  training  problems  should  be  developed  and  made 
widely  visible  throughout  the  potential  user  community. 

User  concerns  about  the  what,  where  and  how  of  CBI  are  by  no 
means  unfounded.  A  recent  report  by  Advanced  Technology  (1985) 
presents  an  extensive  review  of  currently  available  products, 
sources  and  methods  in  CBI  and  a  description  of  present 
technology  applications.  This  "overview"  of  existing  technology 
requires  three  volumes  (over  600  pages)  and  some  800  references 
to  deal  almost  exclusively  with  Air  Force  requirements  and 
applications.  Montemerlo  and  Tennyson  (1976)  present  over  4000 
references  prior  to  1975  in  the  general  context  of  instructional 
systems,  of  which  over  200  deal  specifically  with  CBI.  Kearsley 
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(1983),  in  his  book  on  CBI  method  selection,  provides  180 
references.  The  review  conducted  for  the  present  study,  to  be 
described  later,  examined  over  1600  citations  in  various  areas 
of  CBI  technology.  The  sheer  magnitude  of  available  methods  and 
the  massive  literature  on  CBI  in  its  various  forms  is  burdensome 
to  the  professional  in  the  area  and  almost  certainly 
overwhelming  and  impenetrable  to  the  informed  layman. 

As  Funaro  and  Lane  (1985)  point  out,  a  critical  factor  in 
overcoming  lags  in  technology  applications  is  specific 
intervention  into  the  technology  introduction  and  transition 
process.  With  respect  to  CBI  in  the  military  environment,  there 
has  been  for  some  time  an  awareness  of  the  need  to  achieve 
better  use  of  the  capability  offered  by  the  technology.  Pohlman 
(1984)  and  Dallman,  et  al.  (1983)  looked  at  CBI  software 
multi-use  potential,  support  and  maintenance  from  a  joint 
service  perspective,  and  described  requirements  for  a  software 
"clearinghouse”  function  within  the  Department  of  Defense 
(DOD) .  The  Defense  Science  Board  (DSB)  on  Training  and  Training 
Technology  (OUSDR&E,  1983)  recommended  a  series  of  actions  to 
improve  the  realization  of  CBI  payoff,  including  accelerated 
development  of  the  technology  and  a  focus  on  technology 
demonstrations.  In  1984,  in  part  as  a  response  to  these  and 
numerous  other  training  issues,  the  Office  of  the  Secretary  of 
Defense  established  the  Defense  Training  Data  and  Analysis 
Center  ( TDAC )  in  Orlando. 

The  TDAC  is  a  DOD-level  organization  charged  with  supporting 
systematic  use  of  training  technology  within  the  DOD  and  with 
providing  to  the  services  data  and  assistance  on  evaluation  of 
training  effectiveness.  Among  the  much  broader  functions  of  the 
TDAC  is  the  tracking  of  training  technology  developments  within 
DOD  to  determine  best  use  of  the  technology  and  to  help 
potential  users  in  sorting  out  those  methods  most  appropriate  to 
their  requirements.  Within  that  broader  charter,  an  early 
priority  has  been  a  focus  on  CBI  technology  and  specifically  on 


data  base  issues  associated  with  dissemination  of  technical 
information  to  users.  This  report  describes  one  part  of  a 
coordinated  inhouse/contract  series  of  evaluative  surveys  and 
analyses  directed  toward  obtaining,  organizing,  storing  and 
retrieving  the  data  needed  to  provide  this  decision  support.  In 
addition  to  the  present  document,  findings  of  this  effort  are 
presented  in  Funaro  and  Lane  (1985),  Lane  (1985)  and  Waldrop  and 
Lane  (1985).  While  much  of  the  material  is  reproduced  here, 
these  articles  provide  some  additional  context  and  rationale  on 
selected  topics.  Because  of  the  lack  of  general  availability  of 
the  Lane  (1985)  document,  the  presentation  materials  are 
provided  here  as  Appendix  A. 

The  effort  leading  to  the  insights  described  in  this  report 
focused  initially  on  the  definition  of  current  and  projected 
work  in  research  and  development  (R&D)  for  defense-related  CBI. 
The  general  approach  was  a)  to  provide  an  abbreviated  "snapshot" 
of  present  CBI  capabilities  and  methods,  b)  to  examine  and 
describe  ongoing  R&D  in  terms  of  its  enhancements  of  present 
capabilities,  then  c)  to  discuss  user  issues  for  an  examination 
of  CBI  classification  schema,  and  finally  d)  to  project  the 
outcomes  of  proposed  and  planned  R&D  on  the  availability  of  new 
methods  in  the  near  future.  This  capabilities  definition  is 
aimed  at  providing  a  basic  framework  for  the  eventual 
establishment  of  a  "users"  or  "consumers"  guide  to  CBI 
technology  in  the  form  of  a  data  storage  and  retrieval  system. 

In  such  a  context  it  is  obviously  critical  to  examine  in 
detail  the  characteristics  of  potential  users  of  a  CBI 
technology  data  base  and  their  likely  requirements  for 
information.  As  later  discussions  will  document,  the  term  CBI 
covers  an  extraordinarily  broad  spectrum  of  terms,  categories 
and  classifications,  rarely  with  consensual  meaning  among 
authorities  and  almost  never  among  laymen.  Some  form  of 
assistance  to  the  diverse  user  community  in  accessing  CBI 
information  sources  is  essential,  and  its  absence  is  steadily 
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increasing  in  impact.  Zemke  (1984)  reinforces  this  pressing 
need  for  consumer  guidance,  suggesting  that  increased 
availability  of  commercially  published  CBT  has  changed  the  task 
facing  the  trainer  from  justifying  the  use  of  the  medium  itself 
to  selecting  and  exerting  quality  control  over  courseware, 
whether  off-the-shelf  or  custom  designed.  Funaro  and  Lane 
(1985)  describe  the  data  base  conceptually  as  a  series  of  "file 
drawers"  into  which  the  user  can  look  to  determine  approaches 
that  might  be  appropriate  for  his  or  her  specific  application, 
combined  with  a  way  of  determining  which  file  drawer  to  open  and 
which  file  to  examine  first.  This  guidance  should  be  available 
for  both  state-of-the-art,  established  methods  and  for  those 
technologies  just  emerging  from  R&D. 

Providing  a  "consumer  oriented"  data  base  requires  careful 
attention  to  both  what  should  be  in  the  data  base  (the  contents) 
and  how  it  should  be  organized  and  accessed  (the  structure). 
Although  the  present  effort  necessarily  addresses  both  issues  to 
some  degree,  the  main  focus  is  on  organization,  the  "how  to"  of 
data  structures. 

In  the  process  of  analyzing  and  synthesizing  available 
information  on  past  and  future  CBI,  several  inescapable 
conclusions  emerged.  First,  there  was  no  consensus  (and  very 
little  awareness)  among  system  developers  and  documenters  about 
who  the  "users"  of  CBI  technology  might  be.  The  "intended  user" 
implied  by  context  of  system  descriptions  suggested  a  range  from 
an  unsupported  subject  matter  expert  (SME)  with  a  personal 
computer  to  an  instructional  language  specialist  with  a  full 
courseware  authoring  language  and  a  multi-terminal  mainframe 
system.  Second,  the  data  quickly  confirmed  a  suspicion  that  it 
was  futile  to  attempt  contrasts  of  past,  present  and  future  CBI 
without  some  kind  of  organizing  schema  for  providing 
descriptions  of  various  CBI  system  characteristics.  Fleishman 
(1982)  and  Fleishman  and  Quaintance  (1984)  note  (in  a  slightly 
different  context)  the  importance  of  taxonomic  structures  in 


8 


identifying  and  describing  the  relationships  among  elements. 
Miller  (1962;  1975)  has  emphasized  the  need  for  a  common 
framework  for  both  performance  and  training,  and  has  attempted 
to  develop  training  taxonomies  which  provide  a  behaviorally 
oriented  structure.  In  particular,  Fleishman  and  Quaintance 
systematically  link  the  general  availability  of  taxonomies  and 
classifications  to  both  the  effective  use  of  existing  technical 
information  and  the  informed  shaping  and  guiding  of  future 
inquiry.  It  became  clear  early  in  the  present  effort  that  some 
categorization  of  methods  was  essential  for  the  process  of 
drawing  generalizations  about  the  status  and  future  of  CBI 
alternatives  and  approaches. 

The  remainder  of  this  report  will  provide  an  overview  of  CBI 
technology  status,  trends  and  possible  directions,  and  examine 
ways  of  organizing  and  presenting  information  on  that 
technology.  It  will  address  the  CBI  "user"  issue,  present 
information  from  interviews  with  technology  developers  and 
potential  data  base  users,  and  it  will  examine  existing 
technology  data  bases,  generating  a  "strawman"  schema  for 
classifying  CBI  system  characteristics. 


SECTION  II 
SCOPE 


Objective 

As  previously  described,  the  initial  objective  of  the  CBI 
technology  analysis  and  synthesis  was  to  describe  current  and 
near  future  capabilities  of  CBI  in  the  DOD  environment,  with 
projection  of  the  technology  over  the  next  decade  and 
identification  of  the  most  relevant  applications  of  the  CBI 
technology.  During  the  conduct  of  the  effort,  the  need  for  a 
second  and  eventually  a  third  objective  became  clear  --  to 
define  as  fully  as  possible  the  characteristics  and  data  needs 
of  potential  users  and  to  develop  a  preliminary  schema  for 
organizing  technology  data.  The  resulting  schema  would  provide 
a  departure  point  for  an  eventual  data  base  with  a  computerized 
search  system  to  obtain  up-to-date  information  about  CBI 
technologies.  Data  base  entries  would  contain  descriptions  of 
CBI  system  characteristics  on  a  variety  of  features  important  to 
a  potential  system  user  or  training  decision  maker. 

Framework  and  Context 


Figure  1  illustrates  the  overall  project  framework,  in  the 
context  of  data  base  origin  and  use.  Ideally,  each  of  the  boxes 
in  Figure  1  warrants  an  in-depth  analysis  to  describe  the 
relationship  with  other  aspects  of  the  framework,  as  well  as 
with  aspects  not  included  in  the  illustration.  The  authors 
conducted  a  problem  analysis  on  the  overall  system,  geared 
toward  answering  questions  such  as: 

•  What  information  should  be  in  the  data  base? 

•  Where  does  the  information  originate? 

•  How  is  it  best  classified? 

•  Who  will  use  the  data  base? 

•  How  will  users  retrieve  the  data? 
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DATA  BASE  FRAMEWORK 
FIGURE  1 


A  potential  source  of  confusion  in  dealing  with  CBI  is  the 
multiplicity  of  meanings  subsumed  under  the  CBI  rubric.  These 
definitional  issues  are  the  subject  of  another  study  by  TDAC 
under  its  CBI  program.  For  purposes  of  the  present  analyses,  it 
was  necessary  to  make  a  clear  distinction  between  CBI  in 
Applications  and  computer-supported  CBI  Development  Systems. 
"Applications"  refers  to  an  existing  computer-assisted 
instructional  delivery  system  (CAI)  and/or  an  existing 
computer-managed  instructional  (CMI)  program.  "Development 
Systems"  refers  to  the  multiplicity  of  computer-supported  tools 
that  may  be  used  in  the  development  of  an  application  system. 
The  latter  term  subsumes  such  categories  as  computer-based 
methods  for  training  cost  and  resource  estimation,  courseware 
authoring,  selection  of  instructional  strategies,  and  similar 
functions.  The  technology  review  and  projection  described 
herein  dealt  primarily  with  Development  System  capabilities 
emerging  from  ongoing  or  planned  R&D  efforts  within  the  DOD, 
with  an  additional  focus  on  the  preliminary  classification 
schema  needed  to  create  a  data  base  and  to  tie  current 
developments  to  future  requirements. 

A  further  delimiter  was  necessary  to  establish  boundaries 
for  CBI  which  were  in  line  with  conventional  uses  of  the 
terminology.  Strictly  considered,  CBI  ranges  from  simple 
applications  such  as  computer  presentation  of  textual 
information  to  large-scale  full  system  simulations  (such  as 
aircraft  simulators  with  visual/motion  capability).  (See  Peters 
(1982)  and  Micheli,  Morris  &  Swope  (1980)  for  illustration  of 
the  diversity  of  the  CBI  terminology  umbrella).  To  allow 
reasonable  boundaries  for  the  effort,  such  categories  as  text 
automation,  full  mission  simulators  and  embedded  training 
hardware  (but  not  necessarily  software)  were  excluded  from 
consideration.  Otherwise,  the  working  definition  covered  all 
DOD-related  R&D  work  intended  to  extend  the  state-of-the-art  in 
instructional  development,  delivery  and  management  through  the 


use  of  computer  support  for  some  aspect  of  the  instructional 
process.  Personnel  involved  in  other  TDAC  projects  are 
reviewing  commercially  available  CBI  and  developing  detailed  CBI 
definitions  and  terminology. 

Existing  Data  Base  Utilization 


One  of  the  TDAC  functions  is  to  integrate  and  augment 
existing  operational  data  bases  to  focus  on  analyses  of 
longitudinal  and  historical  training  data.  Existing  sources  of 
data  have  been  designed  for  a  variety  of  other  purposes  and  are 
not  always  fully  compatible.  The  authors  made  no  attempt  to 
gather  the  data  necessary  for  creation  of  an  R&D  data  base. 
With  the  understanding  that  existing  data  bases  will  eventually 
provide  a  departure  point  for  needed  inputs,  this  project 
defined  potential  users  and  sampled  current  R&D  programs,  in  an 
effort  to  recommend  a  classification  schema  for  the  data  base, 
prior  to  extensive  collation  of  existing  data.  The  authors 
examined  one  particular  type  of  information  (CBI  R&D  within  DOD) 
from  the  user's  perspective  toward  the  most  efficient  method  of 
classification.  While  an  extensive  review  and  analysis  of 
existing  information  systems  would  be  a  considerable  effort,  the 
authors  did  attempt  a  cursory  look  at  CBI  R&D  data  bases;  who 
uses  them,  how  and  why  they  use  them,  and  the  pros  and  cons 
associated  with  that  use. 


SECTION  III 
METHOD  AND  APPROACH 


Initial  Technical  Review 


The  project  required  several  different  but  related  efforts, 
the  analysis  of  CBI  R&D  toward  a  user-oriented  data  base  being 
the  overall  goal.  Early  in  the  project,  a  technical  review 
paper  (Funaro  &  Lane,  1985)  provided  a  conceptual  analysis  of 
the  CBI  applications  gap  as  a  problem  in  "acceptance  of 
innovation”,  and  identified  several  specific  actions  required  to 
encourage  application  of  the  technology.  The  article  laid  the 
theoretical  basis  for  the  concept  of  the  CBI  data  base  as  a 
"consumers  guide"  and  for  the  role  of  user  support  systems  and 
organizations. 

Literature  Review 

An  extensive  literature  search  was  used  to  acquire  initial 
information  for  scoping  the  analysis  and  establishing  the 
preliminary  structure  of  the  classification  schema.  A 
bibliographic  search  by  the  Defense  Technical  Information  Center 
(DTIC)  identified  about  1300  defense-related  articles  in  the 
CBI/CAI/CMI  topical  areas  since  1975.  A  search  of  Work  Unit 
Summaries  (DD  1498)  for  a  similar  period  was  used  to  augment  the 
authors'  own  knowledge  of  R&D  activities  within  DOD  and  to 
ensure  that  work  completed  over  the  last  decade  was  not 
overlooked.  Journals  and  other  reference  sources  yielded  in 
excess  of  300  additional  citations.  From  these  initial 
abstracts,  about  350  articles  were  considered  which  dealt 
primarily  with  the  technology  of  CBI  (vs.  applications), 
including  a  number  of  citations  available  from  secondary  sources 
in  prior  reviews.  Approximately  150  articles  were  reviewed  in 
depth;  of  these,  only  about  half  were  directly  and  uniquely 
germane  to  purposes  of  this  analysis.  The  final  reference  list 
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of  90+  citations  from  the  R&D  literature  is  contained  in  Section 
IX:  References. 

Interviews  and  Survey 

The  initial  literature  review  was  supplemented  by 
cross-checks  with  government  project  personnel  working  on 
related  efforts  and  other  specialists  within  the  authors' 
organization,  and  by  informal  discussions  with  active  CBI 
research  personnel  in  a  variety  of  settings,  such  as  conferences 
and  professional  association  meetings.  These  efforts  were 
directed  at  sketching  out  the  preliminary  technology 
classification  schema  to  be  used  in  further  data  collection. 
Formal  "interviews"  were  conducted  with  investigators  from  the 
major  DOD  R&D  agencies  with  active  interests  and  knowledge  in 
CBI  technology  development  and  with  a  selected  cross  section  of 
possible  users  of  the  CBI  data  base  to  expand  and  revise  the 
strawman  schema  (see  Table  1).  Interviews  served  to  obtain  the 
broadest  possible  representation  of  user  concerns  in  the  schema. 

Interviews  were  structured  so  that  each  resulted  in  a 
similar  type  of  information  and  level  of  specificity,  while  at 
the  same  time  allowing  for  flexibility  in  the  discussion. 
Appendix  B  contains  the  interviewer's  outlined  survey  guide, 
used  to  structure  the  interview  so  that  a  certain  set  of 
specific  topics  was  addressed  at  a  minimum,  with  no  constraints 
on  any  other  aspects  of  discussion.  The  structure  consisted  of 
first  establishing  the  type  of  potential  user  being  interviewed, 
e.g.,  a  planner,  researcher,  instructor,  etc.,  followed  by 
discussion  of  critical  aspects  of  the  data  base  structure,  e.g., 
retrieval  systems,  updating  methods,  content,  etc.  The  CBI 
researchers  were  queried  not  only  on  their  current  and  probable 
future  use  of  data  bases,  but  also  on  their  current  technology 
research  programs.  Users  were  sampled  in  both  the  instructional 
development  field  and  in  the  delivery  of  operational  training. 
The  fleet  community  was  included  to  discover  what,  if  any,  use 
they  might  have  for  a  CBI  R&D  data  base,  since  they  are  the 
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TABLE  1 

PARTICIPANTS  IN  THE  SURVEY 


NAME 

James  M.  Bolwerk 

Carl  R.  Driskell 

I 

Dr.  Sharron  P.  Gott 

John  Hayes 

Dr.  Robin  Keesee 

Allen  Marshall 

Dr.  Frank  W.  Schuf letowski 

Dr.  Zita  Simutis 

Dr.  Robert  Wisher 


AFFILIATION 


Commander,  Naval  Air  Force 

Pacific  Fleet 

NAS  North  Island,  CA 

Army  Program  Manager  for 
Training  Devices  ( PMTRADE ) 
Orlando,  FL 

Air  Force  Human  Resources 
Laboratory,  Brooks  AFB,  TX 

Army  Research  Institute 
Alexandria,  VA 

Army  Research  Institute 
Alexandria,  VA 

Air  Force  Human  Resources 
Laboratory,  Lowry  AFB,  CO 

Air  Force  Air  Training  Command 
Randolph  AFB,  TX 

Army  Research  Institute 
Alexandria,  VA 

Army  Research  Institute 
Alexandria,  VA 


Dr.  Wallace  Wulfeck 


Navy  Personnel  Research  and 
Development  Center,  San 
Diego,  CA 


ultimate  recipients  of  the  success  or  failure  of  the  training 
program.  These  interviews  provided  information  on  both  the 
needs  of  the  data  base  user  and  the  information  which  would 
eventually  go  into  the  data  base. 

Data  Bases 


Existing  CB1  data  bases  provided  information  on  what  is 
currently  available  and  how  useful  this  available  information  is 
to  the  community.  The  authors  made  no  attempt  to  critique  the 
existing  data  bases,  but  rather  examined  them  from  the 
perspective  of  their  applicability  to  CBI  technology  R&D.  The 
information  stored  in  these  data  bases  could  eventually  serve  as 
a  part  of  the  inputs  to  the  newly  classified  CBI  R&D  data  base. 
Important  to  this  process  was  how  the  data  were  stored,  updated 
and  retrieved,  and  the  content  of  the  individual  files.  The 
field  interviews  yielded  information  from  users  on  how  often 
they  use  the  features  available  in  current  data  bases,  the 
aspects  considered  beneficial  and  those  not  used. 
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SECTION  IV 
CBI  REVIEW 


Perspective  on  the  Review 

Since  long  before  its  emergence  from  programmed  instruction, 
CBI  and  its  subcomponents  have  generated  an  extensive 
literature.  We  noted  earlier  the  magnitude  and  diversity  of 
CBI-related  studies,  subsuming  as  they  do  progressively  more 
significant  portions  of  the  literature  in  education  and 
instructional  technology,  instructional  psychology,  cognitive 
psychology,  training,  computer  hardware  and  languages, 
simulation  and  gaming,  artificial  intelligence  and  expert 
systems.  There  are  available  many  excellent  histories,  reviews, 
evaluative  summaries  and  technical  forecasts.  These  are  both 
general  in  nature  and  specially  focused,  covering  CBI  theory, 
methods  and  technigues,  guidance  for  use  and  effectiveness  in  a 
variety  of  instructional  contexts.  A  sampling  (with  some 
comments  on  context)  is  given  in  Table  2. 

It  is  not  the  intent  of  this  section  to  provide  still 
another  review  or  to  recapitulate  the  findings  and 
recommendations  of  previous  ones.  As  is  apparent  from  earlier 
sections,  our  concern  is  with  "users."  More  specifically, 
emphasis  is  on  what  the  diversity  of  user  reguirements  (for 
technology  and/or  information)  may  mean  for  a  CBI  data  base  or 
guidance  system.  We  will  thus  focus  predominantly  on  literature 
which  deals  with  the  likely  impact  of  CBI  technology  on  users 
with  specific  selected  functions  in  the  instructional  decision 
making,  development,  delivery  and  management  process. 

Effectiveness  of  the  Technology 

The  value  of  CAI  and  CMI  in  improving  or  maintaining 
training  effectiveness  with  reduced  training  time  is  well 
documented.  In  the  CAI  domain,  Orlansky  and  String  (1979)  and 
Orlansky,  string  and  Chatelier  (1982)  describe  a  review  of  48 
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TABLE  2 
CBI  REVIEWS 


REFERENCE 

Ace,  Sendefer  and  Sciutti 
(1983) 

Advanced  Technology  (1985) 

Back  and  McCombs  (1984) 

Berkowitz  and  O’Neil  (1979) 
Branson  (1984) 

Chambers  and  Sprecher  (1980) 

Dean  (1977) 

Fletcher  (1975a) 

Fullan  and  Pomfert  (1977) 

Gerlach  (1984) 

Hickey  (1975) 

Kearsley,  Hillelsohn  and  Seidel 
(1981) 

Kearsley,  Hunter  and  Seidel 
(1983) 


CONTEXT 

Computer-assisted  ISD 
techniques 

A  compendium  of  CBI  current 
CBI  techniques,  methods  and 
products 

Literature  on  implementing 
self-paced  and  individualized 
instruction,  predominately 
CAI 

Bibliography  of  ISD  including 
CBI 

Business  and  industrial 
applications 

Emphasis  on  hardware/software 
issues 

Benefits  and  risks  of  CBI 

State-of-the-art  survey  and 
critical  issues 

Curriculum  and  instructional 
context 

Instructional  trends  with 
some  reservations  about  CBI 
research 

Survey  of  research  in  CAI 

Business  and  industrial 
training 

Evaluative  survey  of  20  years 
of  CBI  research 


TABLE  2  (con’t) 
CBI  REVIEWS 


REFERENCES 

Kearsley  and  Seidel  (1985) 

Kemner-Richardson,  Lamos 
and  West  (1984) 

Kulik,  Kulik  and  Cohen  (1980) 

Micheli,  Morris  and  Swope  (1980) 

Montemerlo  and  Tennyson  (1976) 
Meierhenry  (1984) 

O'Neil  (1981) 

Stephens  (1979) 

Suppes  (1979) 

Wexley  (1984) 


CONTEXT 

Automation  of  training  and 
training  development 

Resource  guide  for 
instructional  decision  makers 

Effectiveness  in  college 
applications 

Technology  forecast  for  '80s 
and  '90s 

Bibliography  and  review  of  ISD 

Educational  methods  context 

Collection  of  summaries  of 
CBI  status 

Analysis  of  factors  in 
successful  CBI  applications 

Review  of  key  technical  areas 
with  forecasts 

CBI  in  context  of  training 


applications  in  military  settings.  The  typical  application 
achieved  an  effectiveness  equal  or  slightly  superior  to 
conventional  instruction  with  a  time  savings  of  about  30 
percent.  These  findings  were  in  general  confirmed  by  Babbitt, 
Pieper,  Semple  and  Swanson  (1985),  who  performed  a  similar 
comparison  for  studies  from  1977-1984.  While  CBT  continued  to 
show  a  superiority  over  conventional  instruction,  differences 
were  not  as  large  as  those  in  earlier  studies  summarized  by 
Orlansky  and  String  (1979). 

Kulik,  Kulik  and  Cohen  (1980)  performed  a  meta-analysis  of 
59  studies  from  the  college  instruction  domain.  Their  findings 
closely  parallel  those  of  Orlansky  and  String.  CBI  increased 
student  achievement  by  about  0.25  standard  deviations,  while 
reducing  the  time  required  for  instruction  by  36  percent. 
Kulik,  Kulik  and  Bangert-Drowns  (1984)  repeated  the  analysis 
for  elementary  school  applications,  reporting  an  achievement 
increase  of  0.48  standard  deviations. 

Sprecher  and  Chambers  (1980),  in  another  review,  suggest  a 
reduction  in  mean  time  from  CAI  of  50  to  67  percent,  somewhat 
larger  than  that  of  Orlansky  and  String,  but  from  studies 
covering  a  broader  domain.  Montague,  Wulfeck  and  Ellis  (1983), 
in  a  carefully  planned  CAI  experiment,  found  a  decrease  in 
training  time  similar  to  that  indicated  by  Orlansky  and  String 
(about  27  percent),  but  also  reinforced  the  concerns  of 
Orlansky  and  String  that  the  cost  savings  from  CBI  associated 
with  reduced  time  have  not  yet  been  realized  because  of 
lockstep  training  schedules. 

In  general,  where  appropriately  used,  CAI  can  clearly  both 
save  time  to  train  and  increase  typical  levels  of  achievement. 
As  with  most  such  generalizations,  this  is  not  universally 
true.  CAI  can  have  little  or  no  impact  (the  effect  sizes 
reported  are  not  large),  or  even  negative  effect  (see  Gerlach, 
1984).  Kearsley  (1983),  Stephens  (1979)  and  Zemke  (1984) 
provide  excellent  summaries  of  conditions  under  which  CAI 
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applications  ace  most  beneficial.  In  assessing  the  overall 
instructional  value  of  CAI,  it  should  also  be  considered  that 
creative  application  of  CAI  technology  (particularly 
intelligent  CAI)  can  provide  training  for  which  other  methods 
are  inadequate,  too  risky  or  too  costly. 

CM  I  systems  in  various  forms  have  been  in  place  in  military 
settings  for  almost  15  years  (Berkowitz,  O’Neil  &  Wagner,  1980; 
Swope,  Corey,  Evans  &  Morris,  1982),  and  their  strengths  and 
weaknesses  are  well  established.  The  literature  on  CMI  is 
generally  supportive  of  its  effectiveness.  Dollard,  Dixon  and 
McCann  (1980)  describe  a  shipboard  system  for  management  of 
damage  control  training.  They  found  increases  in  student 
thruput,  higher  achievement,  and  reduction  in  administrative 
requirements,  as  well  as  positive  attitudes  from  instructors. 
Swope  et  al.  (1982)  report  an  effective  management  of  student 
flow  in  the  Navy’s  Memphis  CMI  system  and  a  general 
satisfaction  with  system  performance  (see  also  Van  Matre,  1980; 
Federico,  1984).  Johnson,  Graham  &  Carson  (1982)  found 
individualized  CMI  courses  to  be  superior  to  conventional 
instruction  for  relatively  complex  maintenance  tasks,  with  no 
difference  in  later  job  performance. 

Despite  these  and  other  indicants,  broad  statements  about 
CMI  effectiveness  require  qualification  in  the  relationship  of 
CMI  to  the  pacing  of  instructional  sequences.  It  is  likely 
that  the  positive  effects  of  CAI  result  from  its  ability  to 
effectively  individualize  instruction.  CMI,  on  the  other  hand, 
can  act  either  to  enhance  individualization  or  to  suppress  it, 
depending  on  its  design  and  its  implementation.  Micheli  and 
Ford  (1983)  suggest  that  a  lack  of  understanding  of  CMI  by  some 
training  managers  results  in  a  failure  to  capitalize  on  the 
benefits  of  individualized  instruction.  Most  studies  find 
distinct  interactions  between  methods  of  pacing  instruction  and 
student  ability.  Baldwin,  Cliborn  and  Foskett  (1976),  Federico 
(1984),  Hall  and  Freda  (1982),  and  Taylor,  Montague  and  Hauke 
(1970)  report  superior  achievement  from  high  ability  students 
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under  self-paced  instruction,  but  not  under  group-paced 
conditions.  There  are  strong  indications  that  CMI  systems,  to 
be  consistently  effective,  must  accommodate  individual 
differences  in  ability  and  rate  of  learning. 

Even  with  the  positive  track  record  of  CAI  and  CMI,  there 
is  still  uncertainty  about  the  extent  to  which  success  of  the 
technologies  is  due  to  the  computer  media  in  itself.  Kearsley 
and  Seidel  (1983)  indicate  that  most  of  the  potential  benefits 
of  CBI  are  not  inherent  in  the  computer;  they  come  about 
because  of  careful  instruction  and  good  instructors.  Clark 
(1983),  in  a  meta-analysis  of  media  effects,  concludes  that 
there  is  no  competent  evidence  that  media  are  a  causal  factor 
in  learning  and  that  effectiveness  is  due  to  other  aspects  of 
the  instructional  situation.  This  conclusion  is  strongly 
reinforced  by  Gerlach  (1984),  who  suggests  that  the  benefits  of 
CBI  may  be  due  solely  to  systematic  development  of  materials. 
Meierhenry  (1984)  also  points  out  that  media  are  no  more  than 
instructional  delivery  mechanisms,  and  do  not  promote  learning 
other  than  as  agents  of  good  instructional  materials.  Branson 
(1984)  contends  that  effectiveness  of  instruction  is  more 
influenced  by  effectiveness  of  management  than  by  delivery 
media  or  materials,  and  Evans  and  Braby  (1983),  in  a  departure 
from  most  analyses  of  individualized  instruction,  conclude  that 
course  effectiveness  is  predominantly  due  to  sound 
instructional  practices,  rather  than  to  delivery  method  or  to 
method  of  pacing. 

Evolution  and  Revolution  in  CBI 

What  is  now  known  as  CBI,  in  particular  the  CAI  components, 
evolved  rather  gradually  over  several  decades  from  programmed 
instruction  and  teaching  machines  into  a  reasonably 
well-defined  technology  alternative  to  conventional  education 
and  training  methods.  So  long  as  implementation  was 
constrained  to  mainframe  and  minicomputers,  progression  of 
capabilities  followed  this  evolutionary  course  of  steady 
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refinement  and  improvement.  As  with  most  other  technical 
areas,  however,  the  power  and  affordability  advantages  of 
microcomputers  brought  about  an  "overnight"  shift  in  what  was 
possible  with  CAI  and  CMI.  In  particular,  the  marriage  of  CAI 
and  microcomputers  was  an  unusually  fruitful  union,  enabling 
training  approaches  impractical  under  other  approaches.  These 
new  capabilities  materially  steepened  the  growth  curve  of  CBI 
technology,  bringing  about  the  oft-noted  "revolution"  in  CBI. 
Funaro  and  Lane  (1985)  suggest  that  such  revolutionary  changes 
in  capability  tend  to  confuse  the  potential  consumer  of  new 
technology,  with  a  resultant  tendency  for  the  new  entrant  to 
stay  out  of  the  marketplace  (for  videocassette  recorders,  home 
computers,  o£  CBI)  until  the  technology  growth  curve  appears  to 
have  levelled  off. 

The  CBI  revolution  has  brought  about  two  major 
complications  for  the  DOD  user  community.  The  greater 
affordability  of  CBI  has  enabled  and  encouraged  a  wider  range 
of  users  to  become  involved  in  the  selection,  designing  and 
using  of  CBI.  At  the  same  time,  the  difficulty  of  choosing 
between  increasingly  larger  numbers  of  complex  alternative 
approaches  requires  progressively  higher  levels  of  expertise  in 
the  technology.  The  need  of  more  users  for  "data"  about  CBI 
system  characteristics  thus  coincides  with  a  mushrooming  of  the 
amount  and  complexity  of  those  data.  This  impact  is  pervasive 
throughout  the  instructional  delivery  system.  It  affects  the 
decision  maker,  the  instructional  planner,  the  instructional 
designer,  the  instructor  and  the  student  --  individuals  with 
diverse  backgrounds  in  CBI  familiarity  and  dramatic  differences 
in  criteria  for  what  they  expect  from  an  instructional  system. 

A  User-Oriented  Review 

In  the  remainder  of  this  section  we  address  concerns  about 
users  from  the  standpoint  of  the  literature,  what  is  known 
about  users  in  their  interactions  with  instructional 
development  and  delivery  systems.  Section  VI  provides 


perspective  on  these  concerns  from  a  cross-section  of  the  users 
themselves,  and  Section  VII  brings  together  both  kinds  of 
information  in  the  context  of  CBI  data  base  design. 


Implementation  of  CBI  technology  at  various  training  levels 
must  take  into  account  the  sophistication  of  users  and  their 
willingness  and  ability  to  accept  and  support  advanced 
technology  applications.  It  is  thus  important  to  understand 
the  range  of  user  types  and  their  different  needs,  interests 
and  interaction  with  the  CBI  system.  As  Kearsley  (1983) 
indicates,  "...of  all  the  factors  that  affect  the  success  of 
CBT  projects,  user  involvement  is  probably  the  most  important" 
(p.  134).  Montague,  Wulfeck  and  Ellis  (1983)  convincingly 

emphasize  the  importance  of  user  organizational  support  and 
user  participation  in  successful  implementation  of  CBI.  Draxl 
and  Aggen  (1981),  in  examining  a  large  number  of  CBI 
installations,  suggest  that  the  only  common  element  in  those 
that  were  successful  was  the  presence  of  one  or  more  key 
individuals  committed  to  the  success  of  the  effort.  Research 
described  by  Fullan  and  Pomfert  (1977)  shows  the  criticality  of 
implementation  support  above  the  instructional  delivery  level 
and  of  a  willingness  to  "compensate"  for  introduction  of  new 
approaches  by  changes  elsewhere  in  the  system.  Processes  of 
soliciting  user  inputs  and  ultimately  providing  CBI  users  with 
up-to-date  information  are  crucial  aspects  of  achieving  user 
acceptance  and  support  and  are  thus  critical  to  a  successful 
CBI  implementation. 


Discussions  in  Section  VII  identify  some  eleven  different 
classes  of  CBI  "users."  Each  of  these  user  groups  has  unique 
technology  data  base  requirements  and  each  is  differentially 
impacted  by  technology  change.  Serving  user  requirements  for 
CBI  information  thus  involves  the  consideration  of  many 
distinct  descriptors  of  system  characteristics.  These  include 
(but  are  not  limited  to)  such  factors  as  the  following  (adapted 


from  Funaro  &  Lane.  1985): 


•  Transportability  (compatibility  of  language,  hardware, 
dependence,  special  system  features,  or  communication  protocols), 

•  Scope  of  Coverage  (full  system  including  instructional 
development,  delivery  and  student  management  versus  selected 
coverage) , 

•  Characteristics  of  Student  Tracking,  Reporting  and 
Feedback  Systems  (for  CMI), 

•  Background/Characteristics  of  Intended  User  (SME, 
training  manager,  analyst;  knowledge  expected,  special  training 
required) , 

•  Degree  of  System  Tailoring  and  Maintenance  Required 
(off-the-shelf  generic  versus  investment  required  for 
customizing  applications), 

•  Presence/Absence/Characteristics  of  Authoring  Language 
and/or  Authoring  System  (extent  of  support  and  complexity  of 
use) , 

•  Nature  and  Completeness  of  User  Documentation, 

•  Cost  Involved  and  Dependability  of  Cost  Estimation, 

•  Hardware  and  Physical  Interface  Requirements, 

•  Embedding  of  Educational  Principles  (Is  theory  of 
instruction  provided  by  system  or  by  user), 

•  Personnel  Requirements  for  System  Operation 
(operations,  update,  maintenance,  etc.),  and 

•  Availability  to  the  Government  (Proprietary  versus 
Public  Domain) . 

This  partial  list  of  system  descriptors  suggests  the 
comprehensiveness  of  materials  with  which  potential  users  must 
deal.  Discussions  which  follow  focus  on  the  functions  of 
individual  users  within  the  military  training  process,  and 
describe  some  considerations  from  the  literature  about  data 
requirements  which  bear  on  effective  performance  of  their 
functions . 


Instructors.  Much  of  the  early  implementation  of  CBI 
proceeded  without  any  concern  for  the  impact  of  the  methods  on 
the  instructor.  It  was  felt  that  effects  could  only  be  positive 
—  reduced  workload,  less  administrative  responsibility,  more 
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time  to  spend  with  students,  and  so  forth.  There  was  thus  some 
considerable  surprise  (and  probably  some  hurt  feelings)  when  the 
reception  by  instructors  of  this  potential  boon  was  not  only 
frequently  unfavorable,  but  sometimes  hostile  (Fullan  and 
Pomfert,  1977;  Kearsley,  1983).  Sherron  (1976),  in  a  survey  of 
hundreds  of  instructors  and  managers  in  military  CBI 
environments,  found  their  attitudes  to  be  highly  critical  of 
those  CBI  implementations  which  failed  to  consider  impact  at  the 
delivery  level,  particularly  those  which  increased  their 
adminstrative  burden  and  reduced  their  contact  time  with 
students.  Sherron's  data  suggests  not  so  much  an  overt 
hostility  to  the  introduction  of  CBI  and  similar  innovations, 
but  rather  a  strong  skepticism  as  to  the  chances  of  their 
working-level  realities  being  considered  in  design  and 
implementation  of  future  "advanced"  training  systems. 

The  literature  is  conclusive  that  instructor  acceptance  is 
virtually  mandatory  for  successful  implementation  of  any 
training  system,  regardless  of  the  form  of  instructional 
delivery.  It  is  equally  clear  that  acceptance  goes  beyond  a 
merely  passive  reaction;  instructors  should  participate  in  the 
development  of  the  instruction  and,  more  importantly,  should 
understand  the  nature  of  any  new  technologies  and  how  these 
relate  to  their  carrying  out  of  instruction.  Evans  and  Braby 
(1983)  describe  the  inability  or  unwillingness  of  working  level 
instructors  to  deal  with  subtle  distinctions  between  delivery 
technologies,  lumping  together  (and  frequently  ignoring) 
techniques  whose  differences  are  clear  to  specialists  in 
education  and  training  disciplines.  To  most  instructors, 
programmed  texts,  self-paced  CBI,  and  learning  centers  are  all 
the  same  and  all  equivalent  to  "new"  technology.  To  the  extent 
that  one  approach  is  perceived  as  unsatisfactory,  the  "blame"  is 
borne  by  instructional  innovation  in  general.  Back  and  McCombs 
(1984)  make  a  similar  point,  suggesting  that  the  sometimes 
negative  reception  of  self-paced  instruction  is  largely  due  to 
poorly  implemented  CBI,  with  or  without  self-pacing.  Thus,  for 
the  potential  of  innovation  to  be  realized,  instructor  issues 
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should  extend  beyond  the  well-known  requirement  for  instructor 
involvement  in  the  development  process.  They  encompass  a 
specific  attention  to  the  information  needs  of  instructors  as  a 
crucial  factor  in  acceptance.  As  previously  noted,  these  needs 
will  differ  slightly  from  those  of  individuals  elsewhere  in  the 
delivery  process  and  will  likely  involve  information  on  why  a 
certain  delivery  approach  is  used  and  how  to  employ  it 
effectively. 

To  a  significant  extent,  the  use  of  CAI  and  CMI  has  changed 
the  role  of  the  instructor,  and  not  always  consistently. 
Changes  have  been  in  some  cases  negative;  Sherron's  (1976) 
instructors  report  loss  of  student  contact  time  due  to 
administrative  responsibilities.  There  are  directly  opposing 
findings;  in  the  study  reported  by  Dollard,  Dixon  and  McCann 
(1980),  the  introduction  of  CMI  decreased  administrative  burden 
and  increased  opportunity  for  student  contact.  The  long-term 
trends  appear  to  support  the  latter  outcome  (Kearsley,  Hunter  & 
Seidel,  1983).  McCombs  and  Dobrovolny  (1980;  1983),  in  a 
thorough  analysis  of  instructor  functions  under  conventional  and 
computer-based  systems,  describe  6ome  gradual  but  relatively 
permanent  shifts  in  instructor  roles.  They  show  that,  under 
well-designed  CBI,  instructors  move  toward  tutor,  counselor  and 
advisor  functions,  and  away  from  planner,  monitor  and 
diagnostician  responsibilities,  with  these  latter  functions  more 
commonly  performed  by  the  computer.  From  Sherron's  findings 
(1976),  and  those  of  other  research,  by  far  the  majority  of 
instructors  believe  the  counselor /tutor  role  to  be  of  overriding 
importance.  Back  and  McCombs  (1984)  indicate  that  one  of  the 
crucial  factors  in  successful  CBI  implementation  is  a 
well-defined  instructor  role  (understood  by  the  instructor),  and 
suggest  that  criticisms  of  individualized  instruction  are  due 
directly  to  feelings  of  loss  of  control  experienced  by 
instructors  in  a  poorly-designed  CBI  environment.  In  this 
context,  the  shifts  identified  by  McCombs  and  Dobrovolny  (1980; 
1983)  are  clearly  in  the  right  directions  and  should  increase 
positive  perceptions  of  the  methodology  by  instructors.  Other 
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factors  may  also  be  important  for  acceptance.  Francis,  Welling 
and  Levy  (1983)  found  that  instructors’  acceptance  and 
appreciation  of  simulation  CAI  were  directly  related  to  the 
existence  of  alternatives.  Where  actual  equipment  was  limited, 
satisfaction  was  high;  where  it  was  plentiful,  training  on 
simulated  equipment  was  somewhat  less  attractive. 

Providing  instructors  with  information  about  the  training 
system  can  enhance  both  acceptance  and  effectiveness.  McCombs 
and  Dobrovolny  (1983)  show  that,  through  training  and 
orientation,  instructors  are  more  likely  to  understand  and  thus 
to  accept  and  to  make  effective  use  of  computer-supported 
training  systems.  This  understanding  of  the  system  is  in 
addition  to  passive  acceptance,  and  appears  to  be  an  important 
factor  in  achieving  an  integration  of  computer  courseware  and 
CM  I  into  the  overall  instructional  pattern.  Anderson  (1980) 
factor  analyzed  courseware  ratings.  One  of  the  five  major  axes 
of  courseware  desirability  was  ease  of  integration,  the  extent 
to  which  training  materials  could  be  utilized  without  disrupting 
instructional  flow.  Shavelson  and  Winkler  (1982)  conclude  that 
the  best  use  of  CBI  is  in  an  integrated  mode  with  existing 
instruction  in  an  established  instructional  setting,  and  suggest 
that  the  technology  may  not  be  cost  effective  when  used  in  other 
ways.  In  their  view,  instructors  perform  so  many  functions,  and 
are  so  central  to  both  the  management  and  delivery  of 
instruction,  that  "replacing"  instructor  functions  by  computer 
would  be  prohibitively  expensive  and  can  be  rejected  on  those 
grounds  alone,  regardess  of  possible  effectiveness  issues. 

In  sum,  instructors  need  information  which  helps  them 
understand  new  training  materials  and  use  them  as  intended  in  an 
environment  of  heavy  interaction  with  the  student.  Without  such 
inf t -raation,  acceptance  is  likely  to  lag  and  full  effectiveness 
will  not  be  realized. 

Designers  and  Authors  of  Instruction.  Of  all  the  potential 
classes  of  CBI  users,  there  is  the  greatest  lack  of  consensus 


about  the  characteristics  of  those  who  structure  and  author 
courseware  and  course  management  systems.  For  no  other 
participant  in  the  development  and  delivery  process  are  the 
technology  developer's  perceptions  about  user  knowledge, 
background  and  organizational  location  so  diverse.  There  is 
still  implicit  in  most  methods  for  development  of  CBI 
(particularly  CAI)  an  assumption  that  people  who  know  the 
subject  matter  (SMEs),  interacting  with  a  computer  through  some 
software-managed  dialogue,  will  somehow  produce  acceptable 
course  materials.  This  assumption  is  in  essence  an  embedded 
belief  (or  hope)  that  :  SME  +  AUTHORING  LANGUAGE  =  EFFECTIVE 
INSTRUCTION.  While  this  is  likely  true  is  some  cases,  it  is 
incorrect  more  often  than  not,  and  the  training  and  experience 
required  of  designers  and  authors  has  become  a  subject  of 
considerable  concern  in  the  literature. 

We  have  discussed  previously  the  conviction  of  many 
authorities  that  CBI  works  primarily  because  of  the  systematic 
attention  to  instructional  material  it  brings  to  the  development 
process,  and  the  extent  to  which  quality  materials  can  override 
method  of  delivery  (Clark,  1983;  Evans  &  Braby,  1983).  Media 
selection  approaches  were  employed  effectively  prior  to  routine 
use  of  CBI  (Braby,  Henry,  Morris  &  Swope,  1975;  Spangenberg, 
Riback  &  Moon,  1973)  and  the  Systems  Approach  to  Training  (SAT) 
dates  back  to  the  1960's  (Montemerlo  &  Tennyson,  1976).  Given 
such  findings,  it  is  not  surprising  that  the  need  for  CBI  has 
been  questioned  (Gerlach,  1984),  raising  the  query  as  to  whether 
any  other  similarly  systematic  approach  might  not  work  as  well. 
In  that  context,  the  quality  control  of  materials  is  paramount, 
and  some  standardization  of  approach,  if  not  of  language,  is 
important.  Kemner-Richardson,  Lamos  and  West  (1984),  in  their 
CAI  Decision  Handbook,  identify  four  general  classes  of  CBI 
development  "teams"  as  currently  applied.  These  are  (somewhat 
paraphrased) : 

A.  The  Inspired  Programmer-Author  --  The  "team"  consists  of 
an  SME  who  learns  the  languages. 


B.  The  Traditional  Development  Team  —  An  instructional 
designer,  an  SME,  a  programmer,  and  graphics  and  editing 
personnel  as  required. 


C.  The  Computer-Supported  Development  Team  --  Instructional 
designer  and  SME  guided  by  an  authoring  system. 


D.  The  Computer-Guided  Author  --  An  SME  guided  by  an 
intelligent  or  "expert  system"  for  complete  instructional  design. 

A  similar  distinction  is  made  by  Bunderson  (1978),  who 
characterizes  author-supporting  mechanisms  (models,  languages, 
systems)  as  "Artistic,"  "Empirical,"  and  "Analytic,"  according 
to  the  extent  to  which  theory  and  prescriptive  guidance  are 
embedded  in  the  mechanism  and  the  degree  to  which  the  user 
provides  or  is  provided  with  the  structure. 

The  above  articles,  along  with  Montgomery  and  Judd  (1979), 
suggest  a  trend  away  from  the  cross-training  of  a  single 
individual  in  all  the  requisite  skills,  and  toward  the  current 
"traditional"  team,  composed  of  a  number  of  individuals  who 
collectively  provide  the  experience  and  training  required  for 
effective  production.  Such  a  team,  made  up  of  an  SME,  an 
instructional  development  specialist,  and,  depending  on  the 
application,  a  programmer,  would  eventually  evolve  into  a 
smaller  team,  similar  to  type  C  or  D  above.  Team  composition, 
and  ultimately,  whether  or  not  the  team  approach  is  required, 
will  depend  heavily  on  the  progress  of  developments  in  authoring 
languages  and  authoring  systems.  These  serve  to  translate 
subject  matter  knowledge  and  training  objectives  into  courseware 
by  "mediating"  between  the  developer  and  the  desired 
instructional  system  (Schwartz,  1983). 

Authoring  languages  have  existed  in  one  form  or  another 
almost  since  the  beginning  of  CBI,  and  have  evolved  steadily 
along  with  other  aspects  of  the  technology.  Authoring  costs 
represent  the  largest  component  of  CBI  costs  (outstripping 


hardware  for  most  applications)  (Kearsley,  1982),  and  are  at  the 
same  time  the  most  variable  (Zemke,  1984).  Several  excellent 
reviews  of  current  authoring  languages  are  available.  Advanced 
Technology  (1985)  presents  a  compendium  of  languages  with 
references  and  commercial  sources.  Locatis  and  Carr  (1985)  list 
more  than  60  available  authoring  languages.  Rears*  „  (1983) 

provides  some  criteria  for  selection  of  a  language  for  specific 
applications.  Bunderson  (1978)  and  Zinn  and  Bork  (1978)  discuss 
some  theoretical  issues  in  language  development  and  present  some 
of  the  special  concerns  of  language  use  in  the  military  training 
environment.  Reigeluth  (1979)  gives  a  summary  of  the  T1CCIT 
system  and  illustrative  applications.  Himwich  (1977)  presents  a 
comparison  of  PLATO  and  TICCIT  applied  to  the  same  training 
problem;  he  notes  that  while  the  languages  have  different 
strengths  and  weaknesses  and  are  likely  to  be  differentially 
effective  for  varying  applications,  in  no  case  did  the 
courseware  authors  take  full  advantage  of  the  special  features 
in  either  approach. 

The  Himwich  (1977)  study  illustrates  a  key  problem  in  the 
use  of  authoring  languages;  without  appropriate  training  or 
experience  on  the  part  of  the  developer,  even  the  most  primitive 
of  languages  may  offer  capabilities  that  exceed  the  developer's 
skill.  As  Bunderson  (1978)  points  out,  military  systems  are 
particularly  vulnerable  to  this  "production  learning  curve".  He 
cites  four  major  reasons  for  this  vulnerability: 

•  Turnover  in  military  author  or  SME  populations, 

•  Lack  of  formal  author  training, 

•  Lack  of  author  control  over  content,  and 

•  Precedence  of  production  concerns  over  innovation. 

A  particularly  costly  result  of  the  factors  identified  by 
Bunderson  is  the  tendency  by  developers,  in  the  absence  of 
guidance  or  formal  computer  support,  to  produce  computer 
presentations  of  already  existing  course  materials,  the 
so-called  "text  automation".  (We  might  consider  this  the 
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"Uninspired  Programmer-Author".)  As  Montague  and  Wulfeck  (1984) 
(along  with  many  others)  point  out,  "...as  long  as  instruction 
is  text-based  and  teaching  is  to  the  same  specifications,  little 
or  no  difference  is  found"  (between  computer  and  texts).  A  key 
aspect  of  overcoming  text  automation  approaches  is  the 
availability  and  use  of  authoring  systems.  Authoring  systems 
extend  the  capability  of  authoring  languages  by  providing 
structure  and  guidance  in  how  to  use  the  language  for 
instructional  purposes.  They  may  contain  user  training  modules, 
embedded  "expert  systems"  representations  of  how  instruction 
should  be  organized  and  sequenced,  special  "formats”  for 
instruction  presentation,  and  other  features  which  lead  the 
author  through  the  development  process  on  the  basis  of 
instructional  objectives  combined  with  some  embedded  theory  of 
instruction  (see  Bunderson,  1978;  Kearsley,  1984;  Merrill  & 
Wood,  1984). 

Until  very  recently,  use  of  authoring  languages  required 
access  to  a  mainframe  or  minicomputer.  Most  of  the  major 
languages  are  now  available  in  microcomputer  format  with  little 
loss  of  power  (see  lists  in  Advanced  Technology  (1985),  Vol.  II, 
Appendix  F).  The  more  complex  and  sophisticated  authoring 
systems  are  likely  to  follow  the  same  development  path. 
Contrasting  current  capabilities  with  those  of  the  middle  1970’s 
(Berkowitz  &  O’Neil,  1979),  suggests  real  progress  over  that 
period.  Berkowitz  and  O’Neil  described  the  authoring  aids 
available  for  each  phase  of  the  ISD  model,  indicating  the  lack 
of  computer  assistance  for  many  key  aspects  of  the  ISD  process. 
Although  by  no  means  all  of  the  identified  deficiencies  have 
been  resolved,  significant  movement  has  occurred  toward 
resolution  of  most,  and  it  is  probable  that  continued  increases 
in  microcomputer  power  will  expand  these  trends.  Widespread 
availability  of  such  capabilities  will  likely  change  the  nature 
of  the  courseware  author's  functions;  as  with  most  such 
automation,  the  average  quality  level  of  materials  produced  will 
increase,  but  the  innovation  and  imagination  exercised  by  the 
occasional  "artist"  will  be  suppressed.  Although  Avner,  Smith 


and  Tenczar  (1984)  report  increased  productivity  and  improved 
quality  as  a  result  of  authoring  system  use,  tradeoffs  among 
cost,  productivity,  and  quality  factors  are  likely  to  be  unclear 
for  some  time  to  come. 


We  noted  earlier  the  effects  of  microcomputers  in  expanding 
the  user  base  for  CBI.  From  our  examination  of  user  issues,  we 
believe  that  the  potential  community  of  training  developers  is 
materially  broader  than  has  been  previously  suspected.  Johnson 
(1983)  treats  the  problems  encountered  by  training  officers  in 
detached  operations.  Developers  at  the  level  cited  by  Johnson 
are  typically  "disenfranchised"  by  the  increasing  complexity  of 
authoring  aids.  He  shows  that  CBI  can  be  used  effectively  by 
individuals  in  a  shipboard  environment  without  formal  training 
in  either  training  or  computers,  if  information  is  provided  on 
how  to  use  the  technology  and  if  authoring  capabilities 
appropriate  to  simpler  applications  are  made  available. 


Decision  Makers.  Planners  and  Managers  of  Instruction. 
Individuals  charged  with  a)  determining  the  feasibility  and 

affordability  of  alternative  delivery  systems,  b)  selecting  the 
form  and  sequencing  of  instruction  within  that  system,  and  c) 
tracking  both  progress  of  students  and  the  effectiveness  of 
instruction,  need  information  of  a  different  type  and  at  a 

different  level  than  those  in  the  two  preceding  groups. 
Planning  and  management  of  training  requires  an  overall 
perspective  on  technical  and  theoretical  issues  combined  with 
the  realities  of  quality  control,  attainment  of  training 
objectives,  and  student  thruput  (see  Seidel  &  Wagner,  1981). 
Ineffective  management  of  potentially  effective  instruction  can 
negate  much  of  the  advantage  gained  by  careful  instructional 
development;  as  Branson  (1984)  suggests,  "Instructional 

technology  products  don’t  achieve  results.  Only  managers 

achieve  results." 


Only  limited  attention 
literature  to  problems  of 


has  been  given  in  the 
managers  and  planners. 


research 
The  CAI 
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Decision  Handbook  of  Kemner-Richardson,  et  al.  (1984),  is  aimed 
directly  at  instructional  managers.  It  provides  a  well-balanced 
combination  of  information  on  technology  characteristics, 
effectiveness  and  cost.  Portions  of  Kearsley's  guide  to  CBT 
selection  and  implementation  (1983)  deal  with  information 
summaries  relevant  to  the  instructional  manager  role,  in 
particular  the  sections  on  instructional  effectiveness.  He 
provides  guidance  on  the  need  to  rearrange  conventional  training 
schedules  for  greater  flexibility  under  CBI  which  is 
particularly  critical  for  successful  CBI  management.  Kearsley 
(1982)  provides  similar  data  and  guidance  on  costs  of  CBI.  Both 
Kearsley  books,  in  particular  the  1983  guidance  on  selection  and 
implementation  of  CBI,  are  directly  applicable  to  concerns  of 
the  government  or  industrial  decision  maker. 

There  are  striking  consistencies  in  the  literature  with 
respect  to  information  potentially  useful  to  decision  makers  and 
planners.  Stephens  (1979)  collected  from  the  literature  113 
descriptions  of  factors  leading  to  successful  or  unsuccessful 
CBI  implementation.  He  found  that  all  of  these  descriptions 
could  be  grouped  under  three  broad  categories: 

•  Effectiveness  --  Does  the  system  do  what  it  is  supposed 
to  do?  Does  it  meet  objectives? 

•  Ef  f  iciencv  --  Does  it  do  what  il.  does  well?  Does  it 
require  a  resource  expenditure  appropriate  to  its  effectiveness? 

•  Practicality  --  Can  it  work  under  the  unique  constraints 
of  the  organizational  situation?  Is  it  appropriate  for 
implementation  under  the  conditions  prevailing  in  the  intended 
environment? 

Answers  to  these  three  broad  classes  of  questions  would 
clearly  be  of  greater  value  for  decisions  prior  to  selection  of 
a  given  system  than  for  a  post-mortem  evaluation.  For  decision 
making  needs,  information  bearing  on  these  major  issues  should 
be  readily  available  for  each  system  alternative  in  a  form  that 
allows  ready  comparison  among  the  characteristics  and  the 
applications  histories  of  possible  choices. 
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These  questions  can  obviously  be  expanded  almost  infinitely 
to  progressively  finer  levels  of  detail  without  losing  their 
generality-  The  system  descriptors  presented  earlier  in  this 
section  (Funaro  &  Lane,  1985)  can  be  seen  as  special  cases  of 
the  three  classes.  Factors  on  which  data  must  be  provided  to  a 
decision  maker  —  technical  risk,  schedule  risk,  training  time, 
resource  requirements,  cost  and  accuracy  of  cost  estimation  -- 
have  their  eventual  impact  through  one  or  more  of  the  classes. 

Students .  While  the  trainee  is  unlikely  to  be  a  "user"  of 
the  CBI  data  base,  he  (or  she)  is,  along  with  the  instructor, 
hardware,  software  and  courseware,  a  critical  component  of  the 
delivery  system.  As  such,  outcomes  of  student  interactions  with 
other  components  of  the  system  are  likely  to  be  an  important 
part  of  the  data  base.  He/she  can  also  be  a  "designer"  of 
training,  since  self-paced  training  allows  the  student  to 
determine  a  significant  part  of  the  training  paradigm.  Further, 
trainees  are  the  recipient  of  the  training  system,  affected  by 
the  quality  of  materials  and  by  good  or  poor  implementation. 
Poor  CBI  can  significantly  reduce  instructor  contact.  The 
"accultur ization"  of  recruits  in  military  behavior  or  protocol, 
traditionally  provided  by  in-depth  interaction  with  the 
instructor  "role  model,"  is  frequently  seen  as  inadequate  by  the 
operating  units.  The  perceived  culprit  is  the  instructional 
technology  itself,  not  its  implementation,  and  there  is 
continued  risk  of  discarding  both  "baby"  and  "bathwater". 
Readily  accessible  data  about  the  possible  effects  of  new 
technologies  on  the  student  and  the  instructional  situation  may 
be  required  to  avoid  such  reactions. 

As  authoring  capabilities  and  resulting  courseware  become 
more  sophisticated,  there  are  long  term  tendencies  to 
accommodate  to  more  than  one  instructional  pattern  within  a 
training  course.  Understanding  how  students  learn,  and 
isolating  variations  in  the  rates  and  patterns  of  that  learning, 
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has  become  a  major  focus  for  instructional  technology.  This 
upsurge  toward  "cognitive  psychology"  has  been  so  dramatic  that 
Gagne  and  Dick  (1983)  and  Resnick  (1981)  suggest  that  the 
behavioral  research  aspects  of  instructional  technology  have  now 
become  a  subset  of  the  cognitive  psychology  domain.  There  has 
been  for  some  time  an  awareness  that  intelligent  CAI  would 
require  the  ability  to  model  a  student's  learning,  to  predict 
the  events  and  time  course  of  training  (see  Fletcher,  1975b; 
Sleeman  &  Brown,  1982;  Suppes,  1979).  Student  models  are 
required  to  evaluate  alternative  training  approaches  prior  to 
selecting  a  training  configuration  (Bunderson,  1978),  and  for 
refinements  in  individualizing  instruction  (Hall  &  Freda, 
1982).  Accurate  models  are  dependent  on  quantitative 
representation  of  student  learning.  Such  models  both  require 
data  and  provide  it;  properly  constituted,  they  can  take 
unsynthesized  information  from  basic  research  and  translate  it 
into  statements  of  potential  system  effectiveness  directly 
usable  by  training  decision  makers  and  planners. 

As  with  instructors,  the  attitude  of  students  toward  CBT  and 
their  acceptance  of  the  technology  is  well  understood  as  an 
important  implementation  factor  (Back  &  McCombs,  1984;  Stephens, 
1979).  Exactly  how  students  feel  about  CB1  is  not  so  clear. 
Orlansky  and  String  (1979)  found  students  generally  positive  to 
CAI.  Knerr  and  Nawrocki  (1978),  reviewing  the  literature  on 
student  attitudes,  report  a  lack  of  any  pattern  or  consistency 
across  studies.  It  is  likely  that  students'  reactions  are 
situation  specific,  dependent  on  both  the  characteristics  of  the 
delivery  system,  the  instructor’s  attitude,  and  the  instructor's 
use  or  lack  of  use  of  the  system. 

Trends  in  the  Technology. 

Predicting  the  future  course  of  any  technology  area  is 
fraught  with  risks.  As  Suppes  (1979)  points  out,  one  is  likely 
to  be  wrong  both  on  what  will  happen  and  when  it  will  happen, 
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and  the  process  of  putting  such  forecasts  into  print  is  often  a 
forerunner  of  later  embarassment .  Nonetheless,  over  the  course 
of  inspecting  over  1500  CBI-related  items,  some  impressions  of 
direction  for  the  technology  will  of  necessity  emerge.  These 
impressions  are  subjective  assessments  and  integrations,  in 
general  not  attributable  to  any  single  article  or  item  of 
literature,  and  based  primarily  on  perceived  shifts  in  the 
subject  matter  being  pursued  by  workers  in  the  CBI  R&D  field 
over  a  decade  or  more  of  time.  We  will  qualify  our  discussions 
by  insisting  that  they  be  considered  as  trends ,  tendencies  for 
the  technology  to  gravitate  in  certain  general  directions, 
without  any  great  confidence  in  the  time  course  in  which  these 
movements  are  likely  to  occur. 

Despite  more  than  25  years  of  history,  CBI  technology  is 
still  a  young  and  immature  field.  As  previous  sections  note, 
much  has  been  written  about  the  future  of  the  technology  and  its 
potential  for  changing  the  nature  of  instruction.  That  far  too 
little  of  this  potential  has  been  realized  may  be  due  not  so 
much  to  the  inability  of  the  technology  to  deliver  what  has  been 
"promised"  as  to  the  time  frames  in  which  the  contributions  were 
to  take  place. 

At  various  times,  CBI  has  been  presented  as  being  cheaper, 
faster,  more  powerful  than  other  educational  alternatives.  Much 
of  this  has  turned  out  not  to  be  completely  true  -  yet.  But  it 
likely  will  be,  eventually.  A  significant  part  of  what  has  been 
been  expected  has  already  arrived.  Technology  capabilities 
exist  far  beyond  those  presently  utilized.  In  achieving 
utilization,  however,  it  is  important  that  the  capability  and 
affordability  of  technology  should  progress  together. 
Throughout  much  of  the  history  of  CBI,  hardware,  software, 
instructional  theory  and  cost  have  been  chronically  out  of 
phase.  Each  of  these  parts  of  the  CBI  package  has  its  own 
relatively  independent  set  of  drivers  and  tends  to  move  at  its 
own  pace.  It  is  thus  not  the  inability  of  CBI  to  provide  what 
users  want,  but  the  difficulty  encountered  in  getting  the 


technology  act  together.  As  a  result  of  hardware  developments, 
computational  power  is  available  for  great  accomplishments  at 
acceptable  hardware  costs;  the  affordability  of  courseware 
authoring  and  the  sophistication  of  instructional  theory  have 
not,  however,  progressed  rapidly  enough  to  match  the 
computational  capability  and  thus  serve  as  constraints  on 
overall  applicability  of  the  technology.  These  components  will 
in  future  realign  themselves;  in  the  interim,  evaluations  of  CB1 
technology  potential  should  consider  the  developmental  phasing 
problem  in  assessing  the  probable  worth  of  CBI. 

Previous  forecasts  of  directions  in  CBI  technology  have  in 
general  been  remarkably  accurate  as  to  the  shape  of  the  future, 
but  consistently  overoptimistic  as  to  the  timing  of  its 
arrival.  This  is  in  no  way  surprising.  The  movement  of  CBI  R&D 
and  the  extent  to  which  the  technology  is  applied  in  actual 
training  situations  are,  particularly  within  the  military 
environment,  governed  by  the  availability  of  resources  and  the 
willingness  of  decision  makers  to  expend  them.  Both  technology 
development  and  applications  are  driven  by  resource  issues.  The 
ability  of  even  experienced  managers  to  predict  the  vagaries  of 
budgeting,  funding  authorizations,  and  the  future  "popularity" 
of  a  given  technical  area  has  historically  been  little  better 
than  chance.  The  projections  provided  by  Micheli,  Morris  and 
Swope  (1980)  for  the  1985-1995  time  frame  are  thus  far  in  line 
with  trends  but  are  already  beginning  to  slip  in  time  course. 
Estimates  of  technology  availability  by  Sherron’s  (1976) 
training  decision  makers  have  slipped  at  least  5  years  in  the 
decade  since  they  were  made.  In  the  discussions  which  follow, 
we  have  thus  avoided  time  course  predictions  and  attempted  only 
to  detect  broad  tendencies  for  movement  in  certain  directions. 
Given  those  qualifications,  the  paragraphs  below  present,  with 
and  without  attribution  to  other  authors,  our  idiosyncratic  view 
of  the  future  of  CBI  technology. 

Trends  in  Delivery  Systems.  There  is  a  steady  convergence 
of  CAI ,  simulation  and  gaming  technology,  both  in  the  methods 
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employed  and  in  the  recognition  of  common  elements  by  workers  in 
the  various  fields.  The  subtle  boundaries  between  these  areas 
have  all  but  disappeared  in  practice.  We  noted  earlier  that 
most  training  methodologies  in  the  "high-tech"  domains  are  some 
form  of  "computer-assisted  instruction,"  from  large  scale  flight 
simulators  to  simple  text  presentations.  It  is  difficult  to 
distinguish  def initionally  between  a  microcomputer-based 
maintenance  trainer  which  simulates  electronically  the  functions 
of  a  piece  of  actual  equipment  (a  training  device)  and  an 
interactive  tutorial  which  models  the  functions  and  responses  of 
the  same  equipment  (CAI).  This  convergence  has  been  accelerated 
by  the  emergence  of  ICAI  (Groveston,  1982;  Heines,  1983;  Sleeman 
&  Brown,  1982;  Smith  &  Collins,  1983).  There  is  movement  within 
traditional  CAI  toward  models  of  complex  systems  that  can  be 
exercised  in  "what  if"  modes  (Anderson,  1980);  the  resultant 
training  systems  are  not  materially  different  in  form  or 
function  from  simulators  or  what  has  been  typically  isolated  and 
treated  separately  as  "games."  In  particular,  the  use  of  ICAI 
for  training  decision  makers  in  tactics  (Crawford  &  Hollan, 
1983;  Thorndyke  &  Westcourt,  1984),  cannot  logically  be 
separated  technologically  from  "wargaming"  trainers  for  the  same 
purpose . 

Intelligent  CAI  and  expert  systems  have  become  an 
inextricable  part  of  CAI  technology.  They  are  not  going  to  go 
away;  they  respond  to  real  training  problems  and  their  growth 
and  direction  should  receive  explicit  attention.  As  there  is 
more  focus  on  individual  differences  in  learning,  the  "cognitive 
styles"  of  trainees,  this  penetration  of  CAI  by  the  ICAI 
approach  will  increase.  There  is  progressive  emphasis  in  both 
CAI/ICAI  and  in  many  kinds  of  simulation  on  modeling  various 
components  of  the  training  process  --  students,  instructors, 
system  hardware,  opponents  and  expert  strategies  (Richardson, 
1983).  For  some  time  to  come,  ICAI  implementations  will  work 
well  in  "one  shot"  applications  and  stand-alone  systems  (Wolfe  & 
Williams,  1979),  but  assimilation  of  expert  systems  and  similar 
new  mechanisms  for  instruction  into  the  applied  community  will 
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be  slow  due  to  high  cost,  resistance  to  innovation,  and 
difficulty  with  integration  into  existing  delivery  structures. 

In  addition  to  increased  use  of  ICAI  as  an  instructional 
medium,  we  also  see  steady  increases  in  the  use  of  more 
sophisticated  instructional  delivery  hardware.  There  will  be 
greater  use  of  audio  (voice)  and  video  (disc  and  cassette)  and 
more  coordination  between  the  two  presentation  modes.  As  costs 
drop,  greater  emphasis  on  graphics  will  occur,  particularly  in 
training  for  maintenance  and  complex  systems  operation  and  in 
gaming  approaches.  Miniaturization  will  allow  powerful  self- 
contained  delivery  systems  to  be  packaged  in  portable  devices 
that  serve  both  training  and  job-aiding  functions  (Wisher  & 
Boycan,  1984).  As  our  survey  participants  (Section  VI) 
indicated,  however,  drivers  for  development  of  training  delivery 
hardware  will  not  come  from  within  the  CBI  field;  developers 
will  take  what  emerges  from  other  technology  development  areas 
and  adapt  those  capabilities  to  instructional  applications. 

Trends  in  Instructional  Development.  We  believe  that  three 
related  thrusts  will  dominate  instructional  development 
technology  in  the  near  future. 

a)  There  will  be  continued  development  of  intelligent 
authoring  systems  and  languages.  The  gradual  incorporation  of 
expert  knowledge  about  how  to  train  and  how  to  construct 
effective  training  systems  will  significantly  increase  the  power 
of  an  SME  to  translate  his  subject  matter  knowledge  into  quality 
instructional  materials.  Movement  will  be  toward  the 
realization  of  an  "intelligent  partner"  as  an  adjunct  to  the 
instructional  development  team  (Bunderson,  1978;  Schwartz, 
1983).  The  time  course  of  this  development  is  highly  uncertain; 
R&D  in  this  area  is  extremely  expensive  and  progress  will  depend 
directly  on  the  development  resources  made  available. 

! 

b)  The  authoring  systems  concept  will  expand  to  cover 
progressively  larger  portions  of  the  instructional  development 
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cycle.  Linkages  to  the  formal  ISD  process  (Vineberg  &  Joyner, 
1980)  will  increase  and  become  more  deliberate.  Computer-based 
authoring  systems  will  begin  to  function  as  job  aids  for  ISD 
(Berkowitz  &  O'Neil,  1979;  Braby  &  Kincaid,  1981;  Schultz  & 
Wagner,  1981),  replacing  much  current  paper  guidance  on  how  to 
use  the  ISD  process.  Expansions  will  include  emphasis  on 
computer-supported  job  analysis  and  generation  of  task  lists, 
and  other  operations  currently  performed  manually.  New 
capabilities  will  serve  both  in  performing  the  tedious 

bookkeeping  functions  involved  in  ISD  (Ace,  Sendefer  &  Sciutti, 
1983)  and  in  generating  and  conducting  tradeoffs  among 
alternatives  (Kearsley,  1984;  Kearsley  &  Seidel,  1985).  There 
will  be  tendencies  toward  inclusion  of  explicit  instructional 
theory  as  an  embedded  part  of  the  development  package,  with 
greater  attention  to  "pedagogical"  or  "how  to  teach"  issues 
(Merrill  &  Wood,  1984). 

c)  Instructional  development  support  packages  will  become 
both  larger  and  more  sophisticated  (as  discussed  above),  and 
simpler  and  less  computationally  extensive,  with  an  explicit  and 
deliberate  divergence  for  the  two  classes  of  methods.  Simpler 

packages  usable  on  microcomputers  will  emerge,  capable  of 
effective  guidance  for  simpler  applications.  This  trend  is 
already  apparent  in  the  commercial  arena  (see  Locatis  &  Carr, 
1985).  These  trends  will  simultaneously  increase  both  power  and 
accessibility  of  development  software. 

Trends  in  Control  of  Instruction.  As  earlier  sections 
discussed,  much  of  the  CBI  applications  gap  is  attributable  to 
a  generalized  resistance  on  the  part  of  the  ultimate  user 

community  to  innovation  in  the  training  and  education  process. 
While  some  of  this  lack  of  acceptance  is  due  to  a  lack  of 

understanding  of  the  objectives  of  new  methods,  by  far  the 
largest  part  results  from  disagreement  as  to  who  (or  what) 
should  control  the  phasing,  sequencing  and  scheduling  of 
instruction.  As  Kearsley,  Hunter  &  Seidel  (1983)  have  noted. 
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there  is  a  body  of  hard-earned  knowledge  about  how  to  overcome 
this  resistance  to  innovation.  As  this  knowledge  increases  (and 
if  it  is  properly  applied),  we  see  several  distinct  shifts  in 
control  of  instruction  and  instructional  development. 

a)  There  will  be  a  steady  (but  not  rapid)  movement  toward 
understanding  and  acceptance  of  individualized  and  self-paced 
instruction.  This  will  require  an  explicit  education  process 
for  training  managers  and  instructors  in  the  role  of  individual 
differences  in  training  and  the  benefit  of  capitalizing  on  those 
differences  in  terms  of  eventual  operational  readiness.  As  this 
acceptance  evolves,  there  will  be  a  gradual  erosion  of  the  more 
rigid  and  lockstep  scheduling  characteristic  of  the  majority  of 
current  military  training  and  an  accommodation  of  methods  for 
managing  student  thruput  to  the  requirements  of  new  technologies. 

b)  As  ongoing  "cognitive  research"  in  the  CA1  context 
matures  (Gagne  &  Dick,  1983),  there  will  be  a  gradual 
assimilation  of  results  into  mainline  training  and  education 
methods.  Emphasis  on  individual  differences  in  learning  styles 
will  eventually  become  systematized,  and  understanding  of  proper 
use  of  learner  control  and  pacing  knowledge  will  lead  to  "rules" 
for  appropriate  application  of  self  vs.  group  paced  methods.  As 
Hall  &  Freda  (1982)  suggest,  individualized  instruction  works 
better  for  some  outomes  (procedural  knowledge)  than  for  others 
(rule  and  principle  learning).  As  such  distinctions  become 
clearer,  more  effective  use  of  "cognitive  style"  will  be 
possible.  The  increasing  maturity  of  cognitive  research  will 
trigger  a  corresponding  shift  toward  R&D  on  CAI  as  a  method  of 
instruction  rather  than  as  a  delivery  system.  Wexley  (1984)  and 
Gerlach  (1984)  note  the  paucity  of  work  related  to  the  former 
construct,  and  emphasize  the  importance  of  CAI  developing  its 
own  theories  of  instruction,  rather  than  importing  those  which 
were  developed  from  other  non-CAI  experiences  and  contexts. 

c)  Micheli,  Morris  and  Swope  (1980)  forecast  a  shift  toward 
centralized  management  of  instructional  development,  with 


gradual  decentralization  of  implementation  and  with  local 
management  and  control  of  student  flow.  Economic  necessity  will 
to  some  extent  force  continuation  of  that  trend  and  dictate  the 
need  for  multi-use  and  shared  courseware  within  and  across 
services  (Dallman,  et  al.,  1983).  Considerable  ongoing  work  is 
toward  the  direction  of  that  forecast.  Some  current  programs 
discussed  in  Section  VI  provide  for  development  of  instructional 
packages  and  courseware  on  mainframes  with  explicit  provision 
for  downloading  to  microcomputers  at  the  individual  unit  level 
for  implementation. 

Getting  Payoff  from  the  Technology 

We  have  identified  above  an  assortment  of  possible  futures 
for  CBI  technology.  While  the  time  frames  are  nebulous,  we 
believe  that  most  of  those  projections  will  eventually  come  to 
pass  in  the  normal  evolution  of  events.  It  is  also  true, 
however,  that  acceleration  of  the  time  course  for  many  of  these 
trends  is  both  possible  and  beneficial  for  military  training. 
Allocation  of  increases  or  decreases  in  development  resources 
serves  to  some  extent  as  an  intervention  in  the  technology 
evolution  processs.  Decision  makers  should  consider  both 
benefit  and  risk  issues;  there  are  dramatic  differences  in  both 
among  the  potential  trends.  From  our  review,  we  believe  that 
certain  seguencings  of  the  possible  developments  are  more  likely 
to  be  beneficial,  since  some  accelerations  offer  leverage  that 
will  enable  others  to  occur  as  a  matter  of  course.  We  present 
below  some  limited  recommendations  for  emphasis. 

a)  Since  a  major  obstacle  to  implementation  of  existing  CBI 
capability  is  acceptance  by  working-level  users,  specific 
attention  to  that  issue  is  likely  to  provide  both  long  and  short 
term  benefit.  In  our  judgment,  the  key  to  increased  acceptance 
is  information  on  the  workings  and  benefits  of  new  technology, 
readily  accessible  and  presented  in  terms  of  implications  for 
operational  users.  As  later  sections  will  show,  the 
user-oriented  data  base  to  which  this  report  is  addressed  is  a 
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straightforward  mechanism  for  providing  this  information. 
Development  of  the  data  base  should  be  accompanied  by  a  focused 
"marketing"  program  aimed  at  making  its  availability  known  to 
potential  users.  Without  such  a  mechanism  for  disseminating 
information,  new  capabilities  resulting  from  many  of  the  above 
developments  may  have  many  of  the  same  implementation  problems 
encountered  with  existing  methods. 

b)  The  expansion  of  authoring  systems  to  support  the 
broadest  possible  range  of  ISD  phases  and  tasks,  and  the 
addition  of  "intelligence"  to  these  systems  through 
incorporation  of  expert  strategies,  provides  a  means  of  direct 
implementation  of  new  developments  as  they  become  available. 
Such  systems  also  extend  the  capability  of  the  system  developer, 
and  insert  into  the  ISD  process  some  explicit  mechanisms  for 
quality  control,  by  removing  some  of  the  "artistic"  individual 
differences  in  selection  of  approach  and  development  of 
instructional  materials.  Such  "automated"  ISD,  like  the  data 
base  on  technology  capabilities,  allows  the  best  opportunity  for 
incremental  introduction  of  new  methodology  to  the  instructional 
development  community. 

c)  The  development  of  simplified,  low-cost  authoring 
approaches  should  be  considered  for  a  higher  priority  than  at 
present.  We  have  suggested  previously  that  many  of  the 
opportunities  for  CBI  will  occur  in  isolated  units  without 
access  to  large  scale  computation  and  are  relatively 
straightforward  in  nature.  Application  of  complex  authoring 
systems  to  such  requirements  would  represent  technological 
overkill.  Commercial  developers  of  authoring  languages  have 
been  far  more  responsive  than  the  military  R&D  community  to  such 
simpler  instructional  situations.  Adapting  or  making  available 
to  operational  users  some  of  these  less  powerful  courseware 
development  aids  could  significantly  broaden  the  applications 
range  for  CBI.  In  this  connection  also  the  technology  data  base 
could  serve  as  a  means  of  providing  availability  information  to 
potential  users. 
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SECTION  V 

EXISTING  DATA  BASES 

Overview 

Previous  sections  have  established  that  CBI  users  of  varying 
backgrounds  have  requirements  for  different  kinds  of 
information.  To  a  limited  extent,  previous  discussions  have 
indicated  the  general  content  of  the  technology  data  base 
necessary  to  respond  to  these  requirements.  Very  little  of  the 
information  needed  presently  exists  in  a  form  that  can  be 
collated,  organized,  and  inserted  into  a  storage  and  retrieval 
system.  There  are  within  DOD  some  existing  data  bases  and  data 
sources  which  are  accessible  for  some  of  these  requirements, 
i.e.,  descriptions  of  ongoing  R&D  in  CBI  and  related  areas,  and 
research  literature  based  on  DOD-funded  efforts. 

Storage  and  retrieval  systems  described  in  this  section  are 
all  that  were  identified  by  those  persons  interviewed  or  through 
our  own  resources.  If  not  exhaustive,  it  certainly  comprises 
the  majority  of  DOD  systems  for  access  to  CBI  research  data,  and 
is  representative  of  all  the  varieties  of  such  systems. 

While  some  of  the  data  bases  below  are  quite  satisfactory  to 
their  users,  others  are  considered  inadequate  or  lie  dormant 
from  insufficient  dedicated  resources  to  update  and  maintain 
data  currency.  Others  are  not  used  regularly  because  of 
difficulty  with  data  retrieval  or  because  other  methods  of 
obtaining  data  (personal  records,  informal  networks)  are  more 
convenient.  For  the  limited  number  of  systems  identified,  the 
following  paragraphs  discuss  content,  update  methods,  retrieval 
system  and  users. 

MATRIS 

The  Manpower  &  Training  Research  Information  System  (MATRIS) 
works  through  the  DTIC  to  provide  specific  information  on 


certain  DOD  manpower  and  training  (MPT)  program  elements. 
MATRIS  was  created  to  develop  and  maintain  a  decision  support 
system  for  the  Training  and  Personnel  Systems  Technology  (TPST) 
community  which  would  assist  managers,  planners,  and  policy 
makers  at  all  DOD  levels  with  research  and  development 
utilization.  The  users  of  the  MATRIS  data  base  are  primarily 
DOD  decision  makers  with  respnsibilities  in  planning, 
programming,  budgeting,  evaluation,  and  analysis  of  R&D 
programs.  MATRIS  users  can  retrieve  stored  information  which 
includes  identifying  data  (i.e.,  funding,  dates,  researchers, 
and  performing  organizations)  and  narrative  data  (i.e.,  purpose, 
approach,  and  progress).  The  system  is  used  to  identify  current 
research  rather  than  to  retrieve  research  reports. 

MATRIS  products  include  such  data  base  summaries  as  a  budget 
book  for  TPST  R&D  Program  Descriptions,  a  Directory  of 
Researchers,  a  Research  and  Studies  Program  Book,  a  Research  and 
Technology  Work  Unit  Summary  Book,  in  addition  to  various  ad  hoc 
reports,  and  provisions  for  personalized  satellite  data  bases. 
The  users  of  these  capabilities  have  highly  specific  needs  for 
information  on  budget  categories,  work  unit  structures,  goals 
and  objectives,  congressional  categories,  etc.  The  sources  of 
MATRIS  inputs  include  Manpower,  Personnel  and  Training  task  and 
work  unit  summaries  and  program  element  project  data. 

Defense  Technical  Information  Center  (DTIC) 


DTIC,  the  parent  organization  of  MATRIS,  stores  and  provides 
rapid  retrieval  of  all  DOD-funded  R&D  literature  for  DOD 
agencies  and  their  contractors.  Searches  are  available  on  an 
extensive  series  of  keywords  and  are  provided  in  a  variety  of 
formats  including  summary,  abstract  and  complete  documents. 
DTIC  also  has  links,  through  the  National  Technical  Information 
System  (NTIS)  to  the  National  Institute  of  Education's 
Educational  Research  Information  Center  (ERIC)  which  accesses 
other  federally  funded  R&D  documents. 


ARPANET 


The  ARPANET  orginated  in  late  1969  as  an  experimental 
network  and  has  evolved  into  a  successful  means  of  providing 
communication  between  computers.  The  ARPANET  was  divided  in 
1983  into  two  separate  unclassified  networks,  ARPANET  and 
MILNET.  The  ARPANET  serves  primarily  as  an  experimental 
research  and  development  network.  The  MILNET  serves  as  an 
operational  military  network. 

Members  of  the  CBI  research  and  development  community 
contacted  in  the  survey  found  the  ARPANET  well  suited  to  their 
information  needs.  The  electronic  mail  capability  provides  an 
informal  network  for  communication  with  other  researchers, 
bulletin  boards  for  information  notes,  newsletters  for 
information  sharing,  and  many  other  services.  While  the 
information  obtained  through  ARPANET  is  not  summarized  or  placed 
in  context,  it  is  this  type  of  information  exchange  and 
up-to-date  R&D  data  that  the  R&D  community  can  most  effectively 
utilize.  Information  available  in  this  manner  is  somewhat  less 
valuable  to  other  non-specialist  users.  The  ARPANET  allows  the 
user  to  obtain  only  the  data  needed  and  provides  many  menus  and 
choice  points  to  search  the  system. 

In-House  Laboratories  and  Centers 


Some  of  the  potential  data  base  users  have  created  in-house 
information  systems  to  track  projects  and  to  disseminate 
information  within  their  environment.  Review  of  these  systems 
can  provide  a  needed  look  at  how  the  information  is  filed  and 
retrieved  and  the  type  of  utilization  it  receives.  Since  the 
creators  of  these  in-house  systems  are  a  part  of  the  potential 
market  for  the  new  CBI  R&D  data  base,  how  and  why  they  use  their 
in-house  system  provides  a  good  source  for  lessons  learned  for 
creation  of  a  new  system.  These  in-house  information  sources 


could  also  provide  supplemental  inputs  to  satisfy  other  R&D  data 
base  requirements. 

The  Army's  Program  Manager  for  Training  Devices  (PM-TRADE) 
has  a  Mission  Area  Material  Plan  (MMP)  which  combines  five  data 
bases.  The  MMP  is  not  in  itself  a  data  base  for  search  and 
retrieval,  but  rather  an  automated  report  plan.  The  reports 
fulfill  in-house  requirements  and  the  MMP  is  updated  annually. 
This  system  produces  a  System  Summary  Sheet  containing  project 
title,  deficiencies,  descriptions,  dollar  values,  and  other 
needed  information.  A  more  specific  listing,  based  on 
deficiency  number,  is  available  but  it  becomes  classified  when 
the  number  is  inserted.  The  information  is  input  by  the 
principal  investigators  and  each  receives  the  completed 
collection  of  data  in  hard-copy  form.  The  large  document  which 
results  is  cross-referenced  and  projects  are  identified  by 
several  methods  to  increase  usability. 

PM-TRADE  has  also  created  an  in-house  filing  system  for 
publications  on  file.  Each  functional  area  is  responsible  for 
inserting  publications  they  receive  which  may  prove  useful  to 
others.  There  are  currently  over  100  entries,  with  as  many 
waiting  for  secretarial  input.  Each  entry  is  put  on  a  formatted 
sheet  which  requires  pertinent  publication  data,  plus  keywords 
and  an  abstract.  The  keyword  descriptors  are  at  the  discretion 
of  the  person  making  the  entry,  but  consistency  is  maintained 
since  the  principal  investigators  work  in  specified  areas.  The 
system  is  still  undergoing  development  and  is  not  extensively 
utilized . 

The  training  technology  department  of  the  Navy  Personnel 
Research  and  Development  Center  (NPRDC )  maintains  a  data  base 
for  in-house  project  reviews  and  project  status.  The  principal 
investigators  are  responsible  for  keeping  their  projects 
up-to-date.  The  file  was  designed  to  provide  final  year-end 
reports,  with  search  capabilities  for  periodic  status  reviews. 
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The  program  maintains  all  versions  of  the  project  summaries  with 
a  record  of  the  person  making  the  changes.  The  data  base  stores 
information  for  hierarchical  retrieval,  based  on  keywords  and 
level  of  inquiry.  Printing  project  summaries  can  be  specified 
in  a  user  unique  format  or  in  pre-f ormatted  versions,  ready  for 
reports . 

Both  the  Naval  Training  Systems  Center  (NTSC)  (formerly 
Naval  Training  Equipment  Center)  and  the  Army  Research  Institute 
for  Behavioral  and  Social  Sciences  (ARI)  maintain  computer 
supported  R&D  program  data  bases  similar  to  those  described 
above.  These  data  bases  contain  narrative  information  about 
program  purpose,  scope,  and  plans  along  with  milestones, 
schedule  and  progress  information. 

General  Comments 

As  we  have  noted  previously,  there  are  many  different  kinds 
of  users.  All  the  data  bases  reviewed  here  provide  either 
information  on  recently  completed,  planned  or  ongoing  R&D  work 
or  access  to  the  basic  technical  literature  dealing  with  CBI. 
These  are  adequate  to  serve  the  needs  of  R&D  managers  and 
investigators  to  determine  what  work  is  in  progress  in  the  DOD 
labs  or  centers,  but,  in  their  raw  form,  are  not  readily  usable 
by  or  accessible  to  most  other  classes  of  users.  The  undigested 
information  in  technical  reports  and  articles  as  provided  by 
DTIC  or  by  in-house  communication  systems,  while  helpful  to  the 
R&D  user,  is  of  little  or  no  value  to  the  many  other  user 
communities  with  legitimate  needs  for  CBI  information  and/or 
support.  Major  expansion  of  the  concept  of  "data  base"  and  the 
generation  of  considerable  additional  data  is  required  to  serve 
these  latter  users. 


SECTION  VI 
FIELD  COMMENTS 

User  Discussions 

Comments  from  on-site  visits  with  the  data  base  user 
community  ranged  from  feasibility  issues  to  the  specificity 
needed  to  make  the  data  base  useful.  Table  3  summarizes  the 
major  comments  from  the  interviews  concerning  data  base  issues. 
Discussions  below  paraphrase  and  expand  user  comments. 

Overall,  the  community  indicated  that  a  data  base,  if 
designed  with  careful  attention  to  the  needs  to  the  user,  could 
provide  a  powerful  source  of  R&D  technology  project  status  and 
state-of-the  art  summaries.  Potential  users  (predominantly  from 
the  R&D  community)  currently  rely  on  a  network  of  information 
based  primarily  on  personal  contact  and  electronic  mail  links. 
There  was  solid  agreement  that  the  data  base  would,  from  the 
beginning,  need  to  be  as  convenient  and  efficient  as  this 
network  to  replace  it,  and  in  addition  would  have  to  provide 
additional  data  not  readily  available  from  such  a  network.  R&D 
users  felt  their  current  sources  were  adequate,  up-to-date  and 
convenient,  and  made  it  clear  that  the  new  data  base  would  have 
to  exhibit  the  same  characteristics.  One  service  a  new  data 
base  could  provide  is  to  expand  the  user's  current  access  to 
information,  which  at  present  is  limited  to  specific  areas  of 
interest.  Applications-or iented  users  would  like  ready  access 
to  state-of-the-art  summaries  of  capability  without  having  to  do 
literature  searches  and  reviews. 

The  user  community  would  prefer  a  retrieval  system  that 
allows  a  hierarchical  search  mode.  This  system  would  require 
the  user  to  progressively  increase  the  specificity  of  the 
request  to  obtain  the  desired  data.  The  level  of  data  would 
then  be  dependent  upon  how  much  information  is  desired  and  the 
intended  use  once  retrieved.  This  type  of  hierarchical  system 


Feasibility /Need : 

•  The  labs  would  like  to  have  an  active  input  into  the 
data  base  file. 

•  Would  like  to  have  database  use  similar  to  a  management 
information  system. 

•  Success  would  be  feasible  if  data  base  served  to  link 

levels  of  project  progression,  i.e.  6.1  -»  6.2  -*  6.3 

etc . 

•  Could  use  a  good  "map"  of  what  is  out  there  to  serve  as 
a  good  source  to  broker  information  to  facilitate  technology 
utilization. 

•  Must  be  accessible  and  maintainable. 

•  Data  base  use  must  be  mandated. 

•  Need  to  know  where  to  go  and  how  best  to  train  on  the 
new  technology. 

•  Would  like  to  see  data  bases  in  a  common  format  so 
information  obtained  from  each  is  transferable,  similar  in  type 
and  specificity. 

•  Would  be  helpful  to  have  up-to-date  catalog  of  what's 
happening  in  authoring  languages  (for  instance). 

Current  Sources: 

•  Word  of  mouth  network. 

•  ONR  information  base  for  theoretical  research  data. 

•  Electronic  mail  -  ARPANET. 

•  In-house  data  bases,  created  to  track  inside  programs. 

•  Library  of  Congress  and  other  Federal  Documentation 
sources . 

•  Own  reading  and  files. 


TABLE  3  (con't) 

USER  COMMENTS  -  USER  DISCUSSIONS 


Contents /Retrieval: 


•  Retrieval  options  must  be  flexible  for  unique  search 
requirements. 

•  Would  look  for  project  descriptions;  duplicate  efforts, 
complimentary  efforts;  and  the  possibility  of  collaboration. 

•  Need  hierarchical  format. 

•  Information  needs  to  include  the  people  type  issues  and 
theoretical  R&D . 

•  Principles  of  instruction  should  be  included  as  well  as 
modern  learning  theory  and  research  psychology. 

•  Need  to  have  modem  access,  compatible  to  a  PC. 

•  Retrieval  by  keywords  and  subject  matter. 

•  Should  include  a  bibliography  of  reports  from  the 
project. 

•  Don't  try  to  put  everything  into  file,  concentrate  on 
retrieval  flexibility. 

•  Where  to  go  for  information. 

Conf iquration: 

•  Should  be  designed  by  a  group,  not  just  data  automators. 

•  Should  include  all  government  programs. 

•  Should  be  a  server  through  ARPANET. 

•  Take  small  "slice"  first  to  demonstrate  the  value  so 
that  people  will  want  to  be  in  the  data  base. 

Updates : 

•  Must  be  updated  with  interim  information,  do  not  wait 
until  project  is  completed. 

•  POCs  should  be  responsible  for  input,  but  data  base 
maintainer  should  have  someone  to  insure  that  it  is  current. 

•  Updates  of  every  six  months  are  not  often  enough. 

•  Is  going  to  require  someone  to  oversee  maintenance. 
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could  provide  a  summary  page  with  further  information 
available,  if  desired.  The  user  community  would  also  like  to 
have  the  content  include  a  listing  of  relevant  publications 
generated  by  a  referenced  project  or  closely  related  projects. 
This  listing  would  provide  literature  sources  which  might  be 
most  readily  available  to  the  user.  Retrieval  content  should 
deal  with  applications  issues  to  assist  those  working  on 
similar  efforts.  Application  information  would  include  the 
subject  matter  area  on  which  the  technology  is  being  applied, 
the  training  approach,  and  the  types  of  equipment  utilized. 
Points  of  contact  for  obtaining  additional  data  should  also  be 
given. 

One  user  from  the  operational  environment  and  one  with 
training  development  responsibilities  would  like  to  see  the 
data  base  contain  analytical  information  such  as  decision 
models  and  confidence  factors  for  the  availability  of  proposed 
new  methods.  This  configuration  would  make  the  CBI  R&D  data 
base  unique  in  providing  assessments  and  projections  of  the 
state-of-the-art  for  making  decisions  about  applications 
programs.  In  the  users'  views,  this  would  allow  the  earliest 
possible  application  of  R&D  products.  Insertion  of  advanced 
training  technologies  is  frequently  difficult  and  discouraging 
to  potential  new  application  due  to  risks  of  feasibility  and 
uncertainties  about  benefits  and  features.  An  analytical  data 
base  could  help  decision  makers  by  summarizing  technology 
features,  pros  and  cons,  configuration  characteristics,  areas 
of  intended  applications,  probable  costs,  etc.  The  user  could 
tap  the  data  base  when  preparing  a  statement  of  work  or 
conducting  long  range  planning  for  a  potential  CBI  training 
application.  In  addition,  this  summarized  data  on  availability 
should  be  supported  by  one-on-one  follow-up,  if  it  is  necessary 
to  expand  on  search  results.  The  data  base  contents  and 
retrieval  capability  as  envisioned  by  these  users  is  almost 
identical  functionally  to  the  "consumers  guide"  and  "file 
drawer"  concepts  introduced  in  previous  sections. 


Technology  Discussions 


In  addition  to  obtaining  comments  on  the  potential  needs  of 
the  user  community,  the  field  survey  yielded  opinions  on  CBI  R&D 
trends  and  a  sampling  of  R&D  technology  projects  which  would 
eventually  provide  products  for  incorporation  in  the  data  base 
(see  Table  4).  The  majority  of  the  personnel  interviewed  had  a 
predictably  optimistic  view  of  where  CBI  will  be  directed  in  the 
coming  years.  The  information  provided  is  likely  to  be 
accurate,  since  many  of  the  respondents  are  the  R&D  personnel 
who  will  be  performing  the  future  work.  The  objective  of  these 
discussions  of  technology  trends  was  to  obtain  an  idea  of  the 
technology  information  which  the  data  base  must  accommodate  in 
the  near  future. 

There  is  a  general  belief  among  the  R&D  community  that  we 
will  see  more  software  development  work  and  less  hardware.  This 
is  in  response  to  the  need  for  more  flexible  systems  which  are 
transportable  and  compatible  and  for  new  technologies  with  more 
immediate  application  potential.  It  is  felt  that  demands  for 
new  hardware  technology  in  CBI  are  not  as  critical  as  the 
requirement  for  new  means  of  manipulating,  storing,  retrieving 
and  displaying  information.  R&D  for  unique  CBI  hardware  is  not 
a  high  priority.  Several  of  the  participants  indicated  that 
hardware  should  be  derived  from  training  requirements  and  that 
training  should  not  be  retrofitted  to  already  developed 
hardware.  Hardware  developments  should  focus  on  efficient 
methods  of  input/output,  high  resolution  of  displays,  improved 
communications,  etc.  to  maintain  high  quality  CBI.  There 
appears  to  be  some  dissatisfaction  with  the  efficiency  of 
existing  video  output  techniques,  which  have  not  kept  pace  with 
hard  copy  production  technology.  There  are  needs  for  improved 
computer  support  for  CAI  development,  as  well  as  in  training 
delivery  and  management.  There  are  noticeable  shifts  toward 
more  directly  applicable  training  R&D  and  away  from 
high-technology  developments  that  "look  good"  but  lack 
demonstrated  learning  effects.  The  fast  pace  of  CBI  revolution 


TABLE  4 

USER  COMMENTS -TECHNOLOGY  DISCUSSIONS 


Trends : 


•  The  inclusion  of  behavioral  science  within  the  field  of 
computer  science. 

•  Toward  software  development,  not  hardware. 

•  Hardware  technology  aimed  at  increasing  output 

efficiency. 

•  Requirement  to  put  systems  on  a  mainframe  first,  then 
on  micros. 

•  More  focus  on  the  end  product,  not  the  user,  i.e.  the 
emphasis  is  on  short-term  payoffs  for  long-term  solutions. 

•  AI  research  is  stimulating  theoretical  research. 

•  Major  words  that  drive  current  work:  user-friendly, 
artificial  intelligence,  expert  systems,  authoring. 

•  Applications  is  the  "in-thing"  right  now. 

•  Emphasis  to  develop  portable  systems  at  low  cost. 

Major  Projects: 

•  Skills  Training/Theoretical  Research: 

Looking  into  skill  commonality  across  jobs  and  tasks 
and  the  way  people  solve  problems.  Emphasis  toward  developing 
a  new  model  for  training  complex  skills.  Principles  of 
instruction  must  be  married  to  technology. 

•  Advanced  Maintenance  Test  and  Evaluation  Simulation 
System: 

Development  of  modular  technology  applications  to 
maintenance,  both  hardware  and  software.  Develop  panel 
trainers  so  that  government  doesn’t  start  from  the  beginning 
for  each  simulator  contract. 

•  AI  Testbed: 

Application  of  expert  systems  to  perform  instruction 
task  and  to  model  the  instructor,  in  M-16  marksmanship  training. 


TABLE  4  (con't) 

USER  COMMENTS  -  TECHNOLOGY  DISCUSSIONS 


•  Instructional  Support  System: 

System  to  provide  extensive  CMI  support  to  instructors 
in  major  CBI  programs  (75%  dedicated  to  CMI).  Support  for  all 
administrative  functions,  such  as  recordkeeping,  testing 
(creation,  scheduling,  evaluation),  and  data  analysis.  Program 
designed  as  a  menu  driven  system  with  on-line  help  editor 
available  which  can  be  tailored  by  user. 

•  Training  Technology  Application  Programs  ( TTAP ) : 

Overall  Air  Force  program  to  integrate  new 
technologies  through-out  the  Air  Force  training  community. 
Provides  a  process  to  develop,  integrate  and  transfer 
technologies  for  training  models,  prototypes,  networks  and 
management.  Numerous  CBI  specific  projects  under  the  TTAP 
auspices . 

•  Advanced  Training  System  (ATS): 

Developed  to  accomplish  six  training  delivery  tasks: 
information  presentation,  demonstration,  drill-and-practice, 
evaluation,  feedback,  and  remediation. 

•  Personal  Electronic  Aid  to  Maintenance  (PEAM): 

Portable  information  system  for  maintenance 
technicians.  Joint-service  program.  PEAM  is  cartridge  loaded, 
has  large  mass  memory,  is  voice-interactive. 

•  Computerized  Hand-held  Instructional  Prototype  (CHIP): 

Portable,  low-cost,  multi-purpose  CBI  device  with 
audio  and  video  capabilities  for  use  in  nontraditional  training 
environments . 


has  overlooked  the  realities  of  courseware  development  and 
instructional  management,  and  these  areas  are  now  being 
addressed. 

An  additional  requirement  comes  from  the  need  for  flexible 
systems  on  microcomputers,  with  trends  toward  initial 
development  on  a  mainframe  system,  downloading  to 
microcomputers  for  implementation.  This  allows  a  measure  of 
flexibility  during  development  to  review  options,  select  the 
best  alternatives,  and  manipulate  inputs  and  outputs  for  the 
best  instructional  result.  There  is  an  increasing  (and 
unpopular)  trend  toward  seeking  short-term  solutions  for 
long-term  problems.  Short-term  R&D  is  geared  toward  obtaining 
results  that  are  responsive  to  specific  systems  or  operational 
requirements.  This  precludes  much  useful  creative  effort  in 
favor  of  more  quickly  applicable  results,  using  variants  of 
existing  technologies.  There  is  an  almost  universal  feeling 
that  this  trend  threatens  the  health  of  the  future  technology 
base. 

The  popular  "buzz"  words  are  consistent  in  recent  years, 
i.e.,  user-friendly,  artificial  intelligence,  expert  systems, 
and  authoring.  Thrusts  in  the  user-friendly  area  are  designed 
to  develop  methods  of  partitioning  tasks  between  instructor  and 
machine  to  increase  the  efficiency  of  instruction.  Artificial 
intelligence  developments  are  regularly  tracked  for  potential 
applicability  to  training.  AI  and  expert  systems  can  provide 
the  learner  with  an  extremely  dynamic  and  effective  learning 
environment.  Authoring  systems  R&D  can  increase  the  use  of  CBI 
by  making  it  readily  available  to  those  users  who  lack 
programming  expertise.  Emphasis  is  on  increasing  the 
efficiency  of  designers,  authors,  and  planners  through 
automation  of  training  development  support.  Intelligent 
authoring  systems,  natural  language  interaction,  programmer  less 
authoring  and  automated  media  selection  are  examples  of 
authoring  development  support  projects  currently  underway. 


Several  groups  are  concerned  with  expansion  of  cognitive 
theories  about  human  learning  processes  and  their  application 
to  computer-based  learning.  One  goal  is  to  discover  how  people 
approach  problem  solving  and  how  they  interact  with  the 
computer.  Learning  theories  and  student  pacing  techniques  have 
not  kept  up  with  training  delivery  technologies.  A  second  goal 
is  to  determine  the  best  ways  to  use  new  capabilities  for 
modeling  student  knowledge  to  teach  "concept  formation"  and  to 
provide  diagnosis  and  tutorial  assistance. 

Presented  in  Table  4  is  a  sampling  of  major  projects  from 
the  CBI  R&D  community.  These  are  by  no  means  the  only  relevant 
projects,  but  are  illustrative  of  the  type  of  content  that 
should  eventually  be  represented  in  the  data  base.  Each  is 
discussed  briefly  below. 

The  "Instructional  Support  System  (ISS)"  at  AFHRL  (Lowry 
AFB)  is  nearing  completion  and  is  in  preparation  for  on-site 
testing.  The  ISS  was  developed  to  provide  a  set  of  software 
modules  for  supporting  CMI  and  CAI  functions.  This  system  will 
provide  the  instructors  with  powerful  CMI  support.  The  ISS  is 
approximately  75%  dedicated  to  CMI  functions  to  assist  the 
instructor  in  planning  the  course,  scheduling,  tracking, 
pacing,  and  testing  the  student.  In  addition,  the  ISS  provides 
a  flexible  menu  driven  system  to  program  overall  course  plans 
and  to  analyze  resulting  data.  The  ISS  was  developed  on  a 
mainframe  system  and  is  designed  to  be  utilized  on 
microcomputers  when  fielded.  Software  is  written  in  Ada. 

The  "  AI  Testbed"  currently  being  evaluated  at  the  Naval 
Training  System  Center,  in  conjunction  with  the  Army's  PMTRADE , 
is  a  straightforward  application  of  expert  system  technology. 
This  program  is  applied  to  M-16  marksmanship  training  and  is 
designed  to  model  the  instructor.  This  testbed  will  provide 
hard  data  on  the  applicability  of  expert  system  technology  to 
training  devices.  Instructor  models  will  supplement,  rather 
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than  replace,  the  instructor  by  providing  additional  feedback 
on  training  progress. 

One  of  the  projects  at  AFHRL  (Brooks  AFB)  is  "Skills 
Training  and  Theoretical  Research."  It  examines  the  thought 
processes  people  utilize  when  solving  complex  problems.  It 
will  provide  both  new  theoretical  information  and  process 
models  for  the  training  of  complex  skills.  When  developed, 
models  can  be  genetically  applied  across  the  spectrum  of 
training  in  problem-solving  skills. 

An  NPRDC  program,  "Future  Training  Technologies," 
emphasizes  the  construction  of  CBI  systems  for  teaching  the 
operation  of  complex  systems.  It  provides  interactive 
simulations  and  automated  tutorial/explanation  capability. 
NPRDC  has  concentrated  its  efforts  on  decreasing  the  amount  of 
programming  expertise  necessary  to  produce  this  class  of  system. 

The  Air  Force  has  an  ongoing  program  called  the  "Training 
Technology  Applications  Program  ( TTAP ) "  to  facilitate 
development  of  technology  models,  prototypes,  networks, 
management,  integration,  and  transfer  throughout  the  Air  Force 
training  community.  The  TTAP  illustrates  an  organized  method 
of  technology  control  to  develop  instruction,  train  students, 
and  manage  resources.  The  top-down  structured  approach 
employed  in  TTAP  is  viewed  as  crucial  for  a  systems  approach  to 
design  and  development  of  automated  systems  for  ISD,  and  as  an 
ideal  tool  for  designing  information  network  systems.  The 
development  is  aimed  at  a  structured  analysis  and  system 
specification  process  that  will  logically  flow  into  an 
efficient  new  physical  description.  The  approach  includes 
documentation,  proof  of  concept  and  a  requirements  satisfaction 
review. 

Another  Air  Force  effort  is  the  "Assessment  of  New  Training 
Technologies"  Project  sponsored  by  the  Air  Force  Air  Training 
Command.  This  project  provided  insights  into  the  CBT  status 
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within  the  Air  Force  by  assessing  the  state-of-the-art  of 

training  technologies  and  relating  those  capabilities  to 

functional  descriptions  of  current  Air  Force  training.  The 
overall  goal  of  the  project  was  to  provide  a  set  of 

recommendations  designed  to  help  ATC  meet  its  short-  and 
long-term  skill  training  requirements.  The  two-phased  project 
included:  1)  the  identification  of  the  state-of-the-art 

training  technologies  and  methods,  and  2)  forecasts  of  future 
training  technologies.  Findings  are  highly  applicable  to  the 

project  reported  here  and  have  been  referenced  frequently 
(Advanced  Technology,  1985). 


Among  the  ongoing  CAI  efforts  in  the  Air  Force  is  the 
"Advanced  Training  System  (ATS)."  The  ATS  is  designed  to 
accomplish  six  training  delivery  tasks:  information 
presentation,  demonstration,  drill-and-practice,  evaluation, 
feedback,  and  remediation.  A  management  structure  is  also 
included.  The  ATS  is  expected  to  become  operational  in  FY  85. 


SECTION  VII 

CONSIDERATIONS  FOR  A  CBI  DATA  BASE 


User  Issues 

It  was  noted  earlier  that  considerable  discussion  is  devoted 
in  the  CBI  literature  to  serving  the  needs  of  the  technology 
user,  with  virtually  no  attention  given  to  developing  a  clear 
picture  of  the  user  for  whom  that  technology  is  intended.  It 
became  apparent  early  in  our  technology  analysis  that  the 
potential  users  to  which  developers  had  oriented  their  products 
covered  an  extraordinarily  wide  range  of  background,  experience, 
and  formal  training.  The  issue  of  concern  was  not  so  much  that 
the  range  was  inappropriate;  there  are  clearly  many  levels  at 
which  use  of  CBI  technology  can  occur.  The  difficulty  was 
rather  the  apparent  lack  of  awareness  of  the  imprecision 
regarding  the  term  "user",  and  the  consequent  lack  of  packaging 
of  the  technology  to  fit  the  skills  and  experience  of  the 
intended  community. 

The  degree  to  which  the  user  issue  is  satisfactorily 
resolved  for  a  new  CBI  technology  may  well  be  the  most  crucial 
factor  in  determining  the  success  of  that  technology;  it  may 
also  be  the  critical  driver  in  finding  the  best  ways  to  organize 
and  present  a  CBI  technology  data  base.  Typically,  new  CBI 
development  systems  emerging  from  R&D  tend  to  be  extremely 
complex  and  require  considerable  experience  in  subject  matter 
and  sophistication  in  courseware  development.  They  are,  as  a 
result,  costly  to  use  for  instructional  systems  development,  and 
frequently  require  extensive  support  cadres  to  maintain 
continued  quality  instruction.  For  major  military  systems,  such 
personnel  and  funding  requirements  may  be  acceptable,  and  the 
availability  of  front-end  resources  may  allow  realization  of  the 
improvements  in  instructional  quality  and  overall  reductions  in 
life  cycle  cost  possible  with  such  systems.  Such  complex 
computer-supported  development,  however,  may  be  inappropriate 
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for  the  many  smaller  and  less  ambitious  applications  that 
characterize  much  of  military  training  and  constitute  (at  least 
numerically)  the  greatest  market  for  CBI  technology.  Johnson 
(1983)  demonstrates  that  effective  computer-based  training  can 
be  achieved  within  the  constraints  of  providing  a  technology 
package  on  a  personal  computer,  usable  by  a  chief  petty  officer 
with  shipboard  training  responsibility  but  without  formal 
training  in  courseware  authoring. 

The  example  given  by  Johnson  (1983)  is  clearly  at  one 
extreme  of  the  user  spectrum.  At  the  other  is  the  type  of 
large-scale  training  system  defined  previously  which  can  both 
afford  full  CBI  development  and  offer  the  cost-effectiveness 
leverage  to  benefit  from  it.  In  between  these  polar  positions 
fall  the  vast  majority  of  potential  CBI  applications,  and  it  is 
for  these  training  needs  that  considerations  of  cost,  user 
sophistication  and  user  willingness  to  "experiment"  or  take 
risks  with  new  training  approaches  become  paramount.  Section  IV 
provided  a  general  user  overview  and  summarized  insights  from 
the  literature  about  some  selected  user  classes.  In  the  process 
of  that  review,  we  identified  at  least  eleven  possible  types  of 
user  which  technology  developers  might  have  had  in  mind  when 
designing  and  documenting  their  instructional  development 
systems.  These  are: 

1)  Subject  matter  experts  (SMEs)  with  no  formal  training 
in  instruction  or  courseware  development, 

2)  SMEs  with  instructional  experience  but  without  computer 
based  courseware  training  or  experience, 

3)  SMEs  with  formal  training  in  courseware  authoring 
languages , 

4)  Instructional  specialists  without  subject  matter 

expertise , 

5)  Instructional  specialists  with  subject  matter 

background, 

6)  Training  managers  without  instructional  background, 
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7)  Training  managers  with  instructional  experience  but  no 
CBI  background, 

8)  Makers  of  decisions  about  how  much  and  what  kind  of 
training  will  be  provided, 

9)  The  technology  developers  themselves, 

10)  The  "superbeing"  with  all  the  above  skills, 

11)  The  trainee. 

Each  of  these  legitimate  user  classes  benefits  to  some 
extent  from  CBI  technology  and  from  its  improvement.  It  is, 
however,  straightforward  that  such  a  range  cannot  be  served  by 
one  or  even  a  few  different  systems;  complexity  of  a  CBI  system 
and  the  presentation  of  guidance  on  its  use  should  be  targeted 
more  closely  to  user  characteristics  than  has  traditionally  been 
the  case.  Likewise,  the  proposed  data  base  which  provides 
information  about  the  characteristics  and  availability  of 
various  CBI  technologies  should  reflect  in  its  structure  a 
similar  sensitivity  to  the  diversity  of  user  background, 
experience,  and  interest. 

The  Classification  Schema 


Development  of  a  classification  schema  was  addressed  as  an 
effort  to  provide  technological  information  to  these  diverse 
users.  For  this  schema  to  be  a  useful  structure  for  describing 
CBI  system  development  technologies  and  their  capabilities,  it 
must,  at  a  minimum,  provide  for  collecting  two  primary  kinds  of 
information:  1)  what  the  system  is  intended  to  do  (Functions 
Performed),  and  2)  how  it  does  those  functions  (System 
Characteristics).  An  abbreviated  schema  was  prepared  to  use  in 
field  interviews  to  obtain  feedback  from  data  base  users  on  its 
comprehensiveness  and  usefulness  as  a  classification  system. 
The  taxonomic  effort  involved  in  a  full-ncale  classification 
structure  will  be  extremely  complex.  Our  efforts  are  at  best  an 
indicative  start;  we  thus  refer  to  our  "final"  classification 
structure  as  a  "strawman"  schema. 
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The  Functions  of  a  system  describe  which  of  a  number  of 
possible  activities  are  computer  supported  in  the  instructional 
process.  Table  5  shows  the  first  two  levels  of  function 
performed  (the  abbreviated  schema).  The  "final  strawman"  has  a 
third  and  sometimes  a  fourth  level  of  indenture  (a  total  of  68 
entries)  and  is  contained  in  Appendix  C. 

The  Function  breakout  is  an  expansion  of  a  structure  used  by 
Peters  (1982),  modified  by  portions  of  the  categorization  used 
by  Reigeluth,  et  al.  (1982)  and  several  others.  Delivery  of 
Instruction  corresponds  closely  to  the  conventional  uses  of  the 
term  CAI.  Management  and  Support  describes  the  traditional 
content  of  CMI,  although  some  authors  now  distinguish  between 
the  management  of  instruction  and  instructional  administration 
as  separate  uses.  The  third  category  is  less  commonly  included 
in  definitions  of  CBI,  but  is  a  particularly  crucial  component 
of  CBI  technology.  The  presence  or  absence  and  the  nature  of 
computer  support  for  such  functions  as  courseware  authoring  are 
major  features  in  distinguishing  between  alternative  CBI 
development  systems. 

System  Char act eristics /Specifications  can  be  thought  of  as  a 
second  dimension  in  describing  CBI  systems,  with  Functions  and 
System  Characteristics  forming  a  two-way  cross-reference  matrix 
which  could  be  used  for  structuring  collection  of  system 
descriptors  for  the  data  base.  Table  6  presents  the  first  two 
levels  of  indenture  for  the  Systems  Characteristics  outline  (the 
abbreviated  schema);  the  full  structure  of  the  strawman  (a  total 
of  85  entries)  is  contained  in  Appendix  D.  The  top-level 
structure  is  similar  to  one  used  by  Kearsley  (1983),  with 
subcategories  based  on  inputs  from  a  number  of  other  sources. 

The  full  strawman  schema  was  reviewed  by  all  but  three  of 
the  individuals  surveyed.  Comments  were  generally  favorable;  no 
additions  or  changes  resulted  from  the  field  interviews. 
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TABLE  5 

FUNCTIONS  PERFORMED 


Delivery  of  Instruction 

1.  Tutorial-Mode  CAI 

(computer  as  teaching  medium) 

2.  Exploratory-Mode  CAI 

(computer  as  teaching  tool) 

Management  and  Support  of  Instructional  Delivery 

1.  Diagnosis  and  prescription 

2.  Student  testing  and  feedback 

3.  Student  progress  management 

4.  Scheduling  of  instruction 

5.  Recordkeeping  and  reporting 

6.  Continuing  evaluation/update/ 
revision  of  training  system 

7.  System  networking 

Support  for  Training  System  Development 

L.  Management  of  the  development  process 
!.  Authoring  courseware 

l.  Developing  instructional  management  software 

i.  Evaluating  the  delivery  system 


TABLE  6 

SYSTEM  CHARACTERISTICS/ 
SPECIFICATIONS 


A.  Systems  Hardware 

1.  Processor 

2.  Architecture 

3.  Student/Instructor  input  media 

4.  Storage  media  for  training  materials 

5.  Output  media 

6.  Portability 

7.  Facility  requirements 

8.  Performance  capabilities 

B.  System  Software 

1.  Operating  systera(s) 

2.  Student/instructor  interface 

3.  Authoring  language(s) 

4.  Instructional  management 

5.  Programming  languages 

6.  Communications/networking 

7.  Security  provisions 

8.  Licensing/restrictions  on  use 

C.  Courseware 

1.  Instructional  techniques  used 

2.  Transportability/coramonality 

3.  Source(s) 

4.  Type/extent  of  documentation 

5.  Type/extent  of  evaluation 

6.  Extent/nature  of  instructor  interaction 

7.  Revision/updating  provisions 

D.  Costs 


1.  Hardware 

2.  Software 

3.  Courseware 

4.  Personnel  and  staffing 

5.  Operation/maintenance 
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In  addition  to  the  two  major  axes  (Tables  5  and  6, 
Appendices  C  and  D),  there  are  other  dimensions  which  would  be 
of  value  in  locating  technology  within  a  data  base.  One  that 
might  assist  a  user  in  selection  of  a  CBI  approach  is  the 
principal  Areas  of  Application  for  the  products  of  a  system. 
This  structure  is  not  yet  (and  may  never  be)  totally  clear  to 
the  authors;  major  headings  would  be  such  global  training 
applications  as  maintenance,  schoolhouse,  flight  procedures, 
etc.  This  would  provide  the  third  axis  of  a  "taxonomic  cube" 
containing  system  descriptions.  The  structure  of  the  CBI  cube 
is  represented  in  Figure  2.  The  fourth  axis  (or  an  alternate 
third  axis)  would  involve  a  categorization  of  Training 
Outcomes,  the  desired  end-product  of  the  educational  process. 

One  such  outcome  breakout  commonly  in  use  is  that  of  Gagne 
ad  Briggs  (1979),  with  the  familiar  catalog  of  declarative 
knowledge,  procedural  knowledge,  cognitive  strategies,  etc. 
Reigeluth,  et  al.  (1982)  present  a  series  of  categorizations 
from  a  number  of  authors,  along  with  some  supportive  theory. 
Each  categorization  is  compared  with  and  mapped  into  the  Gagne 
and  Briggs  (1979)  list  of  outcomes.  Developing  an  outcomes 
structure  with  sufficient  richness  and  precision  to  describe 

the  diversity  of  CBI  systems  is  an  imposing  challenge.  Most 
existing  categorizations  (including  the  Gagne-Briggs)  are 
sparse  in  description  of  complex  motor  and  perceptual  learning, 
and  the  full  spectrum  of  potential  outcomes  is  not  yet  properly 
covered  in  any  description  of  which  the  authors  are  aware.  It 

is  nonetheless  a  potentially  valuable  axis  of.  the  data  base, 

and  some  version  of  an  Outcomes  outline  should  be  considered  as 
a  data  base  storage  and  retrieval  feature. 

Classification  Schema  and  the  Data  Base 

To  be  of  value,  information  in  a  data  base  must  be 

collected,  coded,  stored,  retrieved  and  presented  in  terms  of 
some  set  of  rules.  These  rules  should  be  generative,  that  is. 
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CONCEPTUAL  STRUCTURES  FOR 
COLLECTION  OF  SYSTEM  DESCRIPTORS 

FIGURE  2 


AXIS 

I  =  FUNCTIONS  PERFORMED  BY  CBI  SYSTEM 

II  =  CHARACTERISTICS  OF  SYSTEM 
lll-A  =  APPLICATIONS  OF  SYSTEM 
lll-B  =  TRAINING  OUTCOMES 


they  should  structure  each  phase  of  information  manipulation 
and  produce  a  consistent  and  orderly  outcome.  The 
classification  schema  presented  above  addresses  primarily  the 
data  collection  phase.  It  is,  in  its  present  formulation, 
conceptual  in  nature;  each  axis  represents  a  broad  range  of 
questions  of  a  similar  type  which  should  be  asked  about  a  CBI 
method  or  system.  The  four  axes  discussed  are  meant  to  prompt 
queries  on  the  following  categories  of  information  about  a 
potential  data  base  item: 

•  What  does  it  do?  (Functions) 

•  How  does  it  do  it  and  what  will  it  cost? 

(Characteristics) 

•  What  does  it  teach?  (Outcomes) 

•  What  kinds  of  training  does  it  support?  (Area  of 

Application) 

The  schema  as  presented  is  thus  an  instrument  for 
structuring  the  data  collection  process,  not  a  means  of  storing 
or  retrieving  information.  Similar,  perhaps  comparable, 
schemata  will  be  necessary  to  organize  other  phases  of  data  base 
operation,  in  particular  the  keying  of  retrieval  menus.  Further 
extensive  examination  of  CBI  taxonomic  issues  will  be  required 
to  develop  and  evaluate  a  structure  appropriate  for  each  phase 
of  data  base  development  and  use. 


SECTION  VIII 
CONCLUSIONS 


The  purpose  of  the  proposed  CBI  data  base  is  to  provide  a 
systematic  and  efficient  way  of  determining  the  capabilities  and 
availability  of  CBI  technologies-  Our  analysis  of  the 
literature  and  the  survey  of  participants  establishes  clearly 
that  there  is  a  need .  and  very  likely  an  ultimate  market .  for 
such  readily  accessible  "digested”  information.  The  "consumers 
guide"  version  of  the  data  base,  similar  to  that  suggested  by 
several  users,  provides  guidance  as  to  a)  what  relevant 
technologies  exist  for  a  given  training  requirement,  b)  which  of 
these  are  most  appropriate  for  that  requirement,  and  c)  which 
could  be  executed  within  available  resources.  A  data  base 
oriented  toward  furnishing  such  guidance  to  a  diversity  of  users 
is  technically  feasible,  but  logi6tically  complex. 

The  data  base  and  its  contents  will  not  generate  itself 
spontaneously.  It  will  require  considerable  up-front  effort  and 
a  continuing  investment  of  resources  to  maintain  usefulness. 
The  need  for  the  data  base  to  be  created,  managed,  interpreted 
and  actively  supported  implies  coordination  by  a  single 
organization  charged  with  the  responsibility  to  make  it  work. 

A  data  base  which  contains  only  "undigested"  information, 
i.e.,  project  characteristics,  work  in  progress  and  research 
citations,  would  be  of  limited  value  to  the  vast  majority  of 
potential  CBI  users.  It  would  support  only  the  CBI  research  and 
research  management  communities,  groups  which  believe  with  some 
justification  that  their  data  needs  are  adequately  met  by 
existing  mechanisms.  While  development  of  the  full-scale  data 
base  is  complex,  it  is  debatable  whether  less  ambitious  attempts 
would  be  worthwhile. 
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-  EXTENSIVE  INSTRUCTOR/STUDENT  INTERACTION 

-  PERMANENT  SUPPORT  CADRE 
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-  INDIVIDUALIZED/TAILORED  TRAINING  LEVERAGE  INCREASES 
WITH  EXPERIENCE 

-  EXISTING  SKILLS  AS  BASIS  FOR  NEW  SKILLS 


APPENDIX  B 


INTERVIEWER'S  GUIDE  FOR  USER  SURVEY 


USER  DISCUSSIONS 


On  the  issue  of  the  feasibility  of  an  R&D  Data  Base: 

Is  information  useful  to  community  possible  to  be 
collected? 

Who  would  be  major  users? 

Update  cycle  recommended,  to  keep  DB  current? 

Do  you  presently  use  an  R&D  Data  Base? 

If  yes,  which  one(s)? 
how  often? 

for  what  types  of  information? 

If  no,  would  you  use  one  if  designed  differently? 

Would  you  like  to  see  another  R&D  DB,  configured  differently, 

which  contains  the  information  from  the  existing  DBs? 

If  no,  why  not? 

If  yes,  what  are  the  critical  areas  for  inclusion? 
what  are  the  critical  issues  in  the  retrieval 
system? 

How  would  you  like  to  search  such  a  DB? 

Oriented  toward: 

Student? 

Instructor? 

Hardware? 

Software? 

Program  element? 

Technology? 

Subject  matter? 

Instructional  technology? 

Instructional  strategy? 

Other . . . ? 


What  do  you  think  will  be  the  major  sources  of  input? 
List : 

How  should  data  be  collected? 

Based  on  taxonomy? 

Volunteer? 

From  program  element  summaries? 

From  other  DBs  and  reformatted? 

Standardized  questionnaire? 

Other? 


AD-A161  776  COMPUTER-BASED  INSTRUCTION  (CBI):  CONSIDERATIONS  FOR  A 
USER-ORIENTED  TECHNOLOGV  DATA  BASE(U)  ESSEX  CORP 
ORLANDO  FL  N  E  LANE  ET  AL.  14  OCT  83  EOTR-83-3 
UNCLASSIFIED  N62269-83-D-B115  F/Q  S/9 


USER  DISCUSSION  (con’t) 


Level  of  specificity  needed? 

General  synopsis  -  Request  information  directly  from 
POC? 

Form  oriented,  i.e.  title,  POC,  PE,  $$,  summary? 

Just  unique  techniques  &  POCs? 

Other? 

Once  in  the  DB,  how  should  the  information  be  organized? 

Review  Function  and  System  Characteristics: 

Adequate  for  technology  issues? 

Any  critical  gaps? 

Organization  adequate? 

Need  another  major  classification? 

Other? 


TECHNOLOGY  ISSUES 


What  do  you  see  as  the  coming  trends? 

What  are  the  areas  of  R&D  intrests  in  terms  of  potential  impact? 
Are  there  any  good  recent  technology  surveys  available? 

Where  would  you  like  to  see  CBI  R&D  go? 

What  are  you  (or  your  group)  working  on  in  the  technology  area? 
(use  interview  survey  list  as  discussion  guide) 


DISCUSSION  FOR  DATA  BASE  CREATORS 


Name  of  Date  Base  - 
Data  Base  Location  - 
Size/Number  of  Records  -  Files  - 
Updating  Policy/Procedures/Tirae  Frame  - 
Source  of  Inputs  - 

Storage  Characteristics  (file  sample)  - 

Retrieval  System/Key  Words 

How  were  initial  inputs  obtained? 

How  was  input/storage/retrieval  system  designed?  (decision 
process) 

Who  are  intended  users? 

How  does  data  base  serve  them  in  particular? 


SECTIONS  I,  II,  III: 
(Considecations  for  discussions) 


Data  collection  on  R  &  D  Project  to  sample  an  overall  view 
of  the  types  of  current  R  &  D  projects  and  their 
characteristics. 

Critical  issues  include:  whether  or  not  the  project  is 
unique  to  subject  matter  or  generalizable  to  other 
applications;  whether  or  not  new  technology  is  being  developed 
or  new  applications/modifications  to  existing  technology  are 
being  studied;  whether  or  not  another  lab,  service,  etc.  will 
be  able  to  use  the  results;  etc. 

Could  project  information  be  stored  in  an  R  &  D  data  base 
and  if  so,  how  would  POC  file  the  information? 

...Does  System  have  Characteristic  (listed)? 

...Is  Charactertistic  just  part  of  system  or  a  major  focus 
under  study? 

...What  is  new  or  ' R&D'  unique  about  Characteristic  under  study? 


CBI  DATA  COLLECTION 
R  &  D  SYSTEM  DATA 
(For  use  with  Sections  I-III) 

System  Name: 

Project  Type: 

Study? 

Prototype  Development? 

Operational  Equip? 

Other? 

System  Status: 

Operational  Date  - 
Percent  Complete  - 
Other  - 

Service  Command: 


POC/Phone: 

Equipment : 

Off-the-Shelf? 

Unique  to  R&D  project? 

Type- 

Input  Device(s): 

Off-the  Shelf? 

Unique  to  R&D  project? 

Type- 

Output  Capabilities: 

System  Designed  For: 

Configuration:  Timeshared  _  Stand-Alone 

Distributed  _ 

System  Maintenance: 

Generalization  Features: 

Unique  Development  Issues: 


SECTION  I  -  ISD  DEVELOPMENT  TOOLS 
SUPPORT  TO  THE  ISD  PROCESS 


ISD  PHASE  SUPPORT  TOOLS 


Not 

Part 

Part  of 
Study 

Focus  of 
Study 

PHASE  I 

Analyze  - 

Analyze  Job 

( 

) 

( 

) 

( 

) 

Select  Tasks  Functions 

( 

) 

( 

) 

( 

) 

Construct  Job  Per£.  Measures 

( 

) 

( 

) 

( 

) 

Analyze  Existing  Courses 

( 

) 

( 

) 

( 

) 

Select  Instruction  Setting 

( 

) 

( 

) 

( 

) 

PHASE  II 

Design  - 

Develop  Objectives 

( 

) 

( 

) 

( 

) 

Develop  Tests 

( 

) 

( 

) 

( 

) 

Describe  Entry  Behavior 

( 

) 

( 

) 

( 

) 

Determine  Sequence  &  Structure 

( 

) 

( 

) 

( 

) 

PHASE  III 

Develop  - 

Specify  Learning  Events/ 
Activities 

( 

) 

( 

) 

( 

) 

Specify  Instruction  Mgt.  Plan 
&  Delivery  System 

( 

) 

( 

) 

( 

) 

Review/Select  Existing 
Materials 

( 

) 

( 

) 

( 

) 

Develop  Instruction 

( 

) 

( 

) 

( 

) 

Validate  Instruction 

( 

) 

( 

) 

( 

) 

PHASE  IV 

Implement  - 

Implement  Instructional 

Mgt.  Plan 

( 

) 

( 

) 

( 

) 

Conduct  Instruction 

( 

) 

( 

) 

( 

) 

PHASE  V 

Control  -  (Evaluation)  - 

Conduct  Internal  Evaluation 

( 

) 

( 

) 

( 

) 

Conduct  External  Evaluation 

( 

) 

( 

) 

( 

) 

Revise  System 

( 

) 

( 

) 

( 

) 
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SECTION  II  -  COURSEWARE  CHARACTERISTICS 


SURVEY  CHECK  LIST 
FOR  CBT  CHARACTER ICS 

ADMINISTRATIVE  RECORDKEEPING  (CMI  Subsystem) 
Student  - 

SSN 

Name 

Class 

Other 


Instructor  - 
SSN 

Name  (s) 

Class 

Other 

Reports  (High  Order)  - 

Individual  (SSN  Scores,  Tests  Items,  etc.) 
Class  (Students  who  passed  (failed)  with 
score/subpart  score 
Other 


Performance  Summaries  - 
Grades 

Exams  Taken/Missed 

Path  Through  Tests  (remediations  by  content  areas) 

Time  Spent  on  Tests 

Other 


Automated  Remediation  - 

Instructor  Notification 

Student  Branching  (Automatic  to  a  certain  level) 


TESTING 

Automatic  testing  at  certain  content  breaks? 

Immediate  evaluation  against  criteria? 

Remediation  - 

Timeliness 

Level 

Automation 

Construction  - 

Set  or  Pool  to  choose  from 
Variable  based  on  student  performance 
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Section  II 


con*  t 


DATA  ANALYSIS 

Item  Analysis  - 

Distractor  Selected 
Corcect/Incoccect 
Time  Spent 

*  of  Items  &  Content  Area  Remediated 

Class  Analysis 

Overall  results 

Class  standing  relative  to  other  classes 
Content  analysis 


OPERATING  MODES 
Student 

Instructional  Programmer 
Instructor 
Systems  Programmer 
Administrator 
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SECTION  III  -  AUTHORING  CHARACTERISTICS 


AUTHORING  ISSUES 


User  Interface  - 
Menu  Driven 
Program  Language 
Other 


Uniformity  of  User  Interaction  - 
Sign-on 
Saving  files 
Keystrokes 
Other 


On-line  Documentation  - 
Helps 
Prompts 

Archive  and  Back-up  Facilities 


On-line  branching 

Specification  while  inputting 

Tracking  of  branches 

Other 


On-line  Input  Checking  - 

Form  and  syntax  of  entries 
Other 


Debugging  Tools  - 

Correctness  of  user-enter  data  bases 
Other 


Videodisc  Preparation  - 
Frame  * 
Recordkeeping 
Other 
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FUNCTIONS  OF  COMPUTER-BASED  TRAINING 
AND  EDUCATION  SYSTEMS 


FUNCTIONS  OF  COMPUTER-BASED  TRAINING 
AND  EDUCATION  SYSTEMS 


A.  DELIVERY  OF  INSTRUCTION 

1.  Tutorial-Mode  CAI  (computer  as  teaching  medium) 

a.  Information/knowledge  presentation 

b.  Drill  and  practice 

c.  Tutorial  dialogue  (Socratic  ICAI) 

2.  Exploratory-Mode  CAI  (computer  as  teaching  tool) 

a.  Simulation  of  equipment  and  environment  (trainers) 

b.  System/prccess  models  ("what  if"  exercises) 

(1)  Games  (competitive) 

(2)  Data  base  inquiry/interaction 

B.  MANAGEMENT  AND  SUPPORT  OF  INSTRUCTIONAL  DELIVERY 

1.  Diagnosis  and  prescription 

a.  Entry  level  assessment  and  assignment 

b.  Within-course  remediation 

2.  Student  testing  and  feedback 

a.  Test  construction  and  development 

(1)  Item  bank  maintenance 

(2)  Branching  logic 

(3)  Individualization  of  tests 

b.  Testing  of  progress 

(1)  On-line 

(2)  Off-line 

c.  Feedback  to  students 

(1)  Progress 

(2)  Diagnostic  information 

(3)  Next  assignment 

3.  Student  progress  management 

a.  Progress  tracking 

b.  Progress/completion  forecasting 
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4.  Scheduling  of  instruction 

a.  Scheduling  student  entry  and  load 

b.  Scheduling  assignments 

c.  Scheduling  instructor-student  interaction 

d.  Scheduling  instructional  resources 

5.  Recordkeeping  and  reporting 

a.  Reports  for  instructors 

(1)  Class  summaries 

(2)  Individual  student  report/cumulative  history 

b.  Reports  for  administrators/managers 

c.  End  of  training  processing 

6.  Continuing  evaluation/update/revision  of  training  system 

a.  Test  items 

b.  Instructional  materials 

c.  Instructional  mix 

d.  Interaction  of  material/student  aptitude 

7.  System  networking 

a.  Within  system 

b.  To  other  systems 

C.  SUPPORT  FOR  TRAINING  SYSTEM  DEVELOPMENT 

1.  Management  of  the  development  process 

a.  Estimation  of  cost 

b.  Estimation  of  resources 

c.  Generation  of  railestones/schedule 

2.  Authoring  courseware 

a.  Author  training 

b.  Generation  of  course  materials 

c.  Instructional  logic  and  strategies  (sequencing) 

d.  Item  and  response  formatting 

e.  Media  selection  support 

f.  Manipulation  of  tasks/objectives  data  bases 

3.  Developing  instructional  management  software 

a.  Scheduling 

b.  Tracking 

c.  Testing 

d.  Recordkeeping/reporting 
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4.  Evaluating  the  delivery  system 

a.  Test  items 

b.  Effectiveness  of  instructional  materials 

c.  Instructional  mix 

d.  Instructional  level/instructional  complexity 

e.  Interactions  with  student  aptitude 
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APPENDIX  D 


COMPUTER-BASED  TRAINING 
SYSTEM  CHARACTER I ST I CS / SPEC I F I CATIONS 


COMPUTER-BASED  TRAINING 
SYSTEM  CHARACTERISTICS/SPECIFICATIONS 


SYSTEM  HARDWARE 

1.  Processor 

a.  Large  mainframe 

b.  Minicomputer 

c.  Microcomputer 

d.  Combination 

2.  Architecture 

a.  Dedicated 

b.  Timeshared  with  other  functions 
'  c.  Distributed 

3.  Student/Instructor  input  media 

a.  On-line 

(1)  Keyboard 

(2)  Touch  panel 

(3)  Light  pen 

(4)  Mouse/ joystick 

(5)  Voice 

b.  Off-line 

(1)  OCR/machine  scoring  format 

(2)  Card 

4.  Storage  media  for  training  materials 

a .  ROM/RAM 

b.  Disk/diskette 

c.  Videocassette 

d.  Videodisc 

5.  Output  media 

a.  Display/monitor 

b.  Hard  copy 

c.  Disk/diskette 

d.  Tape 

e.  Audio  (including  voice  playback) 

f.  Speech  synthesis 

6.  Portability 

a.  Size 

b.  Power 

c.  Ruggedization 


7.  Facility  Requirements 

a.  Physical  space 

b.  Power 

c.  Cooling/environmental 

8.  Performance  capabilities 

a.  Display/raonitor 

(1)  size 

(2)  Resolution/pixel  addressability 

(3)  Color 

(4)  Text/graphics 

b.  Number  of  terminals  supported/supportable 

c.  Maximum  response  time 

d.  Random-access  storage 

(1)  Available 

(2)  Required  by  system 


B.  SYSTEM  SOFTWARE 

1.  Operating  system(s) 

2.  Student/Instructor  interface 

3.  Authoring  language(s) 

4.  instructional  management 

5.  Programming  languages 

6.  Communications/networking 

7.  Security  provisions 

8.  Licensing/restrictions  on  use 


C.  COURSEWARE 

1.  Instructional  techniques  used 

2.  Transportability/comraonality 

3.  Source(s) 

a.  User  developed 

b.  Standard  commercial 

c.  Custom  developed  commercial 

d.  Public  domain 

4.  Type/extent  of  documentation 

5.  Type/extent  of  evaluation 

6.  Extent/nature  of  instructor  interaction 

7.  Revision/updating  provisions 


COSTS 

1.  Hardware 


a. 

b. 


Base  system 

Expansion  (per  station) 


V.  \\V. 
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2. 


4. 

5. 


Software 

a.  Base 

b.  Expansion  (per  station) 


Courseware 

a.  Development  (per  instruction  hour) 

b.  Updating/maintenance 


Personnel  and  staffing 


Operation/maintenance 

a.  Clock  time  basis 

b.  Instructional  time  basis 
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